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ABSTRACT 


Title  of  Dissertation:  Understanding  implementation  success  of  electronic  disease  surveillance 

systems  in  Peru:  A  grounded  theory  study 

Carmen  C.  Mundaca.  Doctor  of  Public  Health  Candidate,  2013 

Dissertation  directed  by:  Ronald  Gimbel,  PhD 

Interim  Chair,  Biomedical  Informatics 
DoD  Patient  Safety  &  Quality  Research 

As  technology  becomes  more  accessible,  many  developing  countries  are  turning  to 
implementing  electronic  systems  to  enhance  their  disease  surveillance  capabilities  and  move 
toward  complying  with  the  International  Health  Regulations.  The  purpose  of  this  study  is  to 
contribute  to  the  evidence  base  of  electronic  disease  surveillance  systems  implementation  in 
developing  countries  by  building  a  framework  that  will  identify  key  elements  in  this  process  and 
serve  as  guidance  to  implementers.  This  study  reflects  findings  associated  with  three  aims:  a)  a 
review  of  the  scientific  and  gray  literature  on  electronic  disease  surveillance  systems 
implementation  experience;  b)  an  expert  consensus  workshop  on  the  same  topic;  and  c)  the 
development  of  a  disease  surveillance  system  model  using  the  qualitative  approach  of  Grounded 
Theory  and  informed  by  qualitative  data  provided  by  34  surveillance  stakeholders  in  Peru.  The 
researcher  triangulated  findings  from  each  of  the  three  aims  to  validate,  complement  and 
generate  a  revised  model.  She  developed  this  model  based  on  the  confluence  of  several  factors  at 
different  levels.  The  model  indicated  the  institutional  administrative  level  as  a  starting  point  for 
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disease  surveillance  system  implementation.  High-level  administrators'  commitment  to 
implement  and  sustain  the  surveillance  system  by  enacting  legislation  and  allocating  resources 
proved  to  be  key  to  the  successful  implementation  of  such  a  system.  National  and  regional 
surveillance  teams  performed  the  critical  functions  of  disease  surveillance-related  training, 
monitoring,  supervision,  and  feedback.  The  teams  addressed  the  critical  needs  of  surveillance 
staff,  contributing  to  the  motivation  of  reporting  staff  and  ensuring  high  performing  staff  at 
surveillance  sites.  Finally,  surveillance  staff  found  it  motivating  to  observe  the  outcomes  of  their 
work,  including  appreciating  the  usefulness  of  the  data  they  reported  in  guiding  decision-making. 
As  a  result  of  their  reporting  efforts,  they  saw  disease  outbreaks  averted  and  preventive  actions 
implemented  in  an  effort  to  improve  the  health  status  of  the  population.  The  positive  impact  of 
disease  surveillance  on  population  health  served  as  a  primary  motivator  for  surveillance  staff  and 
stakeholders  to  continue  their  work.  The  model  as  a  result  of  this  study  will  potentially  serve  as  a 
useful  tool  in  facilitating  the  implementation  of  electronic  disease  surveillance  systems  in 
country  contexts  similar  to  that  of  Peru. 
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CHAPTER  1:  BACKGROUND  AND  LITERATURE  REVIEW 


The  severe  acute  respiratory  syndrome  (SARS)  epidemic  and  the  influenza  A 
(H INI)  pandemic  have  reinforced  the  value  of  disease  surveillance  worldwide  (9;  35). 

I'  *l|  . 

Subsequent  to  the  events  surrounding  SARS,  the  public  health  community  in  developed 
countries  made  greater  efforts  to  improve  public  health  capabilities,  investing 
increasingly  large  sums  of  money  to  enhance  their  existing  surveillance  systems  and  to 
develop  complementary  early  outbreak  detection  tools  (7;  12;  71;  82). 

Developing  countries  not  only  have  been  the  source  of  new  diseases,  but  also  the 
majority  of  the  global  infectious  disease  burden  remains  concentrated  among  the  poorest 
20%  of  the  world's  population  (10;  34;  45).  Unfortunately  due  to  competing  priorities, 
many  of  these  countries  have  neglected  disease  surveillance,  and  thus  have  invested  little 
in  building  an  infrastructure  to  support  it  (89).  Surveillance  research  in  resource  limited 
settings  has  typically  been  focused  on  specific  diseases  of  interest  to  funders  and 

.  Pr: 

implemented  through  laboratory-based  projects  of  limited  duration  ( 14).  An  Institute  of 
Medicine  (IOM)  workshop  summary  entitled,  “Achieving  Sustainable  Global  Capacity 
for  Surveillance  and  Response  for  Zoonotic  Diseases",  recognized  the  need  to  build 
capacity  for  sustainable,  long  term  surveillance  and  response  in  countries  with  a  high 
likelihood  of  emerging  infections  (8).  Workshop  participants  acknowledged  the  existence 
of  current  disease  surveillance  networks  and  reporting  systems  worldwide.  They  noted 
the  great  effort  required  to  enable  these  systems  to  continually  distribute  timely,  high 
quality  information  (8).  Such  efforts  have  included  the  integration  of  surveillance  and 
health  information  systems  (74). 
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Electronic  surveillance  systems  are  important  tools  that  are  recognized  for  their 
increasing  importance  in  disease  detection  and  monitoring,  improved  quality  and 
effectiveness  of  data  collection,  and  data  analysis  and  information  dissemination  (75). 
Experience  has  shown  that  despite  potential  benefits,  not  all  implementation  efforts 
introducing  new  technologies  in  healthcare  settings  are  successfully  adopted  (79).  Failure 
occurs  in  up  to  sixty  to  seventy  percent  of  technology  implementation  efforts,  leading  to  a 
substantia)  waste  in  investment  (3).  Moreover,  a  dearth  of  literature  exists  regarding  the 
experience  of  implementing  surveillance  systems  in  resource-limited  settings  (20).  There 
are  few  evaluative  case  studies  from  which  one  could  estimate  the  success  and  failure 
rates  of  public  health  surveillance  systems  (59).  While  some  evidence  exists  on 
sophisticated  disease  surveillance  systems  implemented  in  developed  countries  (11;  43; 
92),  such  experiences  with  disease  surveillance  systems  cannot  be  directly  generalized  to 
the  developing  world  due  to  differences  in  demographic,  sociological,  and  cultural 
characteristics;  the  political  and  organizational  context;  and  resource  availability 
including  technology  (41;  64;  75).  Furthermore,  the  guidelines  developed  by  the  Centers 
for  Disease  Control  and  Prevention  (CDC)  for  Evaluating  Public  Health  Surveillance 
Systems,  while  offering  a  useful  resource  that  describes  the  performance  attributes  of  a 
good  surveillance  system,  fall  short  in  offering  direction  on  weighing  the  importance  of 
these  attributes.  More  importantly,  these  guidelines  do  not  provide  indicators  useful  in 
operationalizing  such  attributes  around  implementation. 

Due  to  the  limited  availability  of  published  literature  on  the  implementation  of 
electronic  disease  surveillance  systems  in  developing  countries,  the  researcher  developed 
a  systematic  review  of  the  literature  to  identify  the  potential  implementation  attributes 
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associated  with  health-related  programs  in  developing  countries,  The  decision  to  use 
health-related  programs  as  a  proxy  to  diseases  surveillance  system  was  based  on  the  fact 
that  disease  surveillance  systems  are  a  key  component  of  the  public  health  programs  of  a 
country  or  an  organization  (17).  Therefore,  the  principles  applied  to  the  planning  and 
implementation  of  diseases  surveillance  systems  are  similar  to  those  used  for  health- 
related  programs  (66). 

The  methodology  of  this  literature  review  is  described  in  Appendix  A.  A  total  of 
32  studies  were  identified  for  inclusion  in  this  systematic  review  from  a  combined  pool  of 
9,785  abstracts  (i.e.,  888  abstracts  from  the  initial  search  and  8,897  abstracts  from  the 
second  search).  A  total  of  9,695  abstracts  were  excluded  from  initial  review  due  to  their 
lack  of  relevance  or  inability  to  meet  established  inclusion  criteria.  To  be  included  in  the 
systematic  review,  studies  had  to  meet  the  following  criteria:  a)  be  indexed  to  one  of  the 
selected  information  sources,  b)  be  written  in  English,  c)  be  focused  on  a  developing 
country(ies),  and  d)  must  have  addressed  the  implementation  of  a  specific  health-related 
program.  There  were  no  restrictions  on  date  of  publication,  specific  research  methods, 
focus  of  intervention,  or  outcome  measure(s)  addressed.  Opinion  or  commentary 
manuscripts  were  excluded  only  if  they  did  not  meet  the  four  criteria  outlined  above 
From  the  abstract  review,  a  total  of  90  manuscripts  were  selected  for  full-text  review.  A 
data  extraction  instrument  developed  for  this  study  (Appendix  B)  was  applied  to  all  90 
manuscripts,  leading  to  the  exclusion  of  58  manuscripts. 

The  primary  outcomes  of  interest  in  the  review  were  the  potential  factors  affecting 
the  implementation  of  health-related  programs  in  developing  countries.  Although  this 
literature  review  did  not  specifically  address  disease  surveillance  systems  implementation 
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(only  three  papers  focused  on  disease  surveillance  (46;  69;  91)),  the  researcher  believed 
that  these  findings  might  still  be  applicable  in  initially  guiding  this  research. 

The  researchers  independently  reviewed  the  32  selected  studies  and  grouped  their 
salient  findings  into  primary  categories  and  subcategories.  The  four  primary  categories 
that  emerged  from  the  review  included:  Program  Design,  Accountability  and  Leadership, 
Engagement,  and  the  Environment  (See  Figure  1).  Each  of  the  four  categories  identified 
will  be  discussed  below,  and  the  detail  of  the  each  primary  category  is  described  in 
Appendixes  C,  D,  E,  and  F. 


Figure  1:  Concepts  identified  in  a  systematic  literature  review  on  factors  associated  with 
successful  implementation  of  health-related  programs  in  developing  countries. 


Program  design 

The  program  design  category  included  potential  implementation  factors  that 
directly  addressed  the  design  or  structure  of  each  health  program.  Within  this  category 
the  researcher  included  two  sub-categories,  which  were  training,  and  program  flexibility. 


4 


The  researcher  learned  that  the  authors  of  12  papers  who  addressed  program 
design  considered  training  to  be  an  important  factor  in  health  program  implementation. 
The  format  of  a  given  training  session  was  deemed  important  by  the  authors  of  two 
papers.  Specifically,  several  brief  formal  training  sessions  for  health  workers  were  found 
to  be  more  effective  than  a  single  multi-day  training  session  (18).  A  cascade  approach  to 
training  was  preferred  (18),  and  small-group  training  discussions  were  favored  over  large 
gatherings  (28).  Several  authors  emphasized  the  preparation  and  provision  of  training 
manuals,  guidelines,  and  other  technical  documents  to  participants  (13;  25;  28;  57;  70). 
The  provision  of  these  materials  were  believed  to  be  useful  in  building  trust  in  the 
program  (28)  and  aided  in  the  adaptation  of  larger  programs  to  local  conditions  (13;  57; 
70).  Some  programs  leveraged  existing  mass  communication  for  their  training  activities 
in  a  given  country  or  region  in  an  effort  to  reduce  cost  and  broaden  exposure  (1;  22).  Of 
the  32  papers  reviewed,  the  authors  of  four  papers  reported  efforts  to  measure  the 
association  with  or  influence  of  training  on  health  program  outcomes.  Findings  associated 
with  one  study  revealed  improved  performance  (related  to  likelihood  of  missing  data  in 
an  information  system)  as  a  consequence  of  training  (36),  while  findings  from  other  three 
studies  identified  no  significant  relationship  between  training  and  desired  program 
outcomes  (e.g.,  reduced  mortality,  outpatient  utilization)  (2;  13;  48). 

Program  flexibility  was  a  notable  design  issue,  especially  when  applied  to 
information  systems.  Some  systems  integrated  local  data  into  a  larger  system  to  provide 
an  understanding  of  risk  and  change  (27),  while  two  other  systems  allowed  for  local 
adaptation  of  the  software  and  available  data  to  accommodate  need  (36;  50).  Still  other 
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systems  were  designed  to  fit  the  medical  environment  (57),  or  were  built  using  a  staged 
“bottoms-up”  approach  based  on  local  need  (3 1 ). 

Another  aspect  of  program  flexibility  related  to  strategies  for  engaging  health- 
related  program  beneficiaries.  In  some  programs,  interventions  were  packaged  differently 
in  an  effort  to  tailor  them  to  the  target  audience  (18).  while  others  allowed  for  flexibility 
by  bringing  the  program  into  local  communities  and  complementing  local  infrastructure 
(83). 

Accountability  and  Leadership 

The  health  program-related  literature  that  was  categorized  as  accountability  and 
leadership  included  potential  implementation  factors  that  addressed  leadership  and 
accountability  in  the  areas  of  process  and  performance.  The  researcher  identified  three 
subcategories  that  included  accountability  policies,  program  leadership,  and  monitoring 
&  evaluation/supervision. 

The  authors  of  seven  papers  highlighted  the  presence  or  absence  of  accountability 
policies  as  potential  factors  for  consideration  in  program  implementation.  They  cited  a 
need  for  governmental  legal  and  regulatory  support  for  health-related  program 
implementation  as  important  (39).  Health  programs  suffer  if  they  lack  government 
capacity  to  support  accountability,  lack  authority  to  impose  penalties  on  poor  performers, 
or  organizations  fail  to  address  the  program  in  public  health  policies  and 
recommendations  (46;  91). 

The  authors  of  ten  papers  addressed  the  importance  of  leadership  as  a  potential 
factor  in  program  implementation.  While  the  authors  of  two  papers  discussed  the  need  for 
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well-placed  individuals  to  serve  as  program  coordinators  or  gatekeepers  (3 1 ;  103),  others 
focused  their  attention  on  the  need  for  active  leadership  at  all  program  levels,  including 
the  governmental  level  (6;  25;  26;  44). 

The  need  for  adequate  supervision  in  overseeing  all  aspects  of  a  health  program 
was  addressed  by  the  authors  of  eight  papers  (2;  13;  25;  39;  40;  57;  69;  106).  Likewise, 
the  need  for  a  formal  program  monitoring  and  evaluation  system,  as  well  as  a  responsive 
adaptation  process  to  feedback  received  from  the  monitoring  and  evaluation  was  also 
addressed  (2;  40;  57).  As  there  was  no  single  monitoring  and  evaluation  process 
recommended  for  all  programs,  variation  in  the  processes  occurred  among  the  programs  . 
What  was  consistent  was  the  need  to  limit  the  number  of  core  indicators  included  in  the 
evaluation  in  an  effort  to  avoid  overwhelming  the  predominantly  low-resourced  settings 
(13;  1 8;  3 1 ;  50).  However,  one  risk  of  limiting  the  number  of  core  evaluation  indicators 
was  that  they  could  be  the  “wrong  indicators,’’  that  did  not  measure  program  success  or 
proved  to  be  unattainable  indicators  given  program  resources  and  setting  (28;  78). 

Engagement 

The  health-related  program  category  of  engagement  included  potential 
implementation  factors  that  addressed  how  programs  connected  with  their  targeted 
audiences  and  other  key  stakeholders.  The  researcher  identified  four  subcategories  related 
to  engagement  that  included  communication,  use  of  incentives,  stakeholder  involvement, 
and  building  local  capacity. 

Communication  as  a  key  concept  surfaced  in  five  of  the  papers  reviewed  as,  a 
potential  implementation  factor.  Internally-focused  communication  factors  related  to 
feedback  mechanisms  in  providing  staff  with  an  understanding  of  their  performance  (69), 
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improving  referral  and  information  feedback  using  simple  tools  (60),  and  the  need  to 
enhance  the  flow  of  communication  among  the  various  organizational  levels  (e.g. 
program,  local,  regional,  national  government)  (25;  78;  91 ). 

The  use  (or  lack  of)  incentives  as  a  mechanism  to  engage  both  target  program 
beneficiaries  such  as  community  members  in  regions  where  the  program  was 
implemented  as  well  as  program  staff  was  addressed  by  authors  of  five  papers.  Two  of 
the  programs  that  employed  financial  incentives  for  both  staff  and  beneficiaries’  for 
program  participation  reported  negative  findings  (60;  78),  specifically  related  to  the 
implementation  of  a  tuberculosis  program  in  each  of  two  developing  countries.  An 
assessment  of  the  tuberculosis  program  found  no  direct  relationship  between  financial 
incentives  and  improved  diagnosis  or  treatment  (60).  In  one  program  that  targeted 
tuberculosis  in  Nigeria,  the  use  of  financial  incentives  was  counterproductive  in  that  it 
established  an  unsustainable  precedent  for  beneficiaries'  participation  (78).  The  use  of 
non-financial  incentives  (e.g.,  food)  for  participation  was  identified  as  an  important 
element  in  three  programs  with  no  negative  results  reported  (40;  83;  106).  In  one  program, 
access  to  diagnostic  facilities  and  educational  programming  was  used  as  incentives  to 
encourage  staff  participation  (57). 

Engaging  key  stakeholder  groups  early  and  frequently  was  cited  as  an  important 
strategy  for  successful  program  implementation  (2;  6;  60).  Interactive  engagement  with 
key  stakeholder  groups  led  to  positive  outcomes  such  as  skill  development  (5 1 ;  54),  local 
ownership  of  key  processes  (51),  motivated  workers  (54),  and  improvement 
modifications  to  the  intervention  itself  (29).  One  major  stakeholder  group  cited  as  a 
primary  target  for  engagement  was  the  Ministry  of  Health  (MOH)  in  each  country  where 
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a  health  program  was  implemented.  Creating  a  role  for  the  MOH  in  program 
implementation  appeared  to  contribute  to  a  sense  of  program  ownership,  or  strong 
government  commitment  (1 ;  25;  54),  a  positive  attitude  among  various  parties  (3 1 );  and 
ultimately  a  positive  impact  on  policy  implementation  (26;  54). 

Developing  local  capacity  can  prove  to  be  fruitful  in  terms  of  program  output  and 
outcomes.  Evidence  suggests  that  developing  local  capacity  can  lead  to  improved  quality 
of  care  (2;  83),  empowerment  of  local  people  and  communities  (27),  local  ownership  of 
the  program  (51),  resource  and  data  sharing  (31),  and  decreased  dependence  on  external 
services  (83). 

Environment 

The  environment  category  included  attributes  of  the  operating  environment  (e.g., 
local,  regional,  national)  of  the  program  to  be  implemented.  It  is  noteworthy  that  while 
the  program  may  have  some  influence  over  its  operating  environment,  some  attributes  are 
beyond  its  control.  The  researcher  identified  five  subcategories  of  environmental 
influences  that  included  political  environment,  social  environment,  human  resources, 
existing  infrastructure  and  capacity,  and  donor/external  support,  each  of  which  will  be 
explained  below. 

The  political  environment  was  an  environmental  factor  in  health  program 
implementation  that  was  addressed  by  the  authors  of  ten  papers.  Five  authors  touted  the 
importance  of  local  and  national  political  support  in  the  coordination  of  program  efforts 
(2;  4;  25;  44;  63).  Other  authors  outlined  requirements  for  gaining  political  support  that 
included,  for  example,  positive  program  results  (4),  timely  data  reporting  (91),  and 
international  partnerships  (78).  In  terms  of  political  environment,  several  authors 
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highlighted  the  challenges  posed  by  fluidity  of  a  political  support  base  and  government 
stability  (106),  actual  emergencies  that  captured  government  attention  (57),  and 
competing  national  priorities  (106). 

The  social  environment  as  an  environmental  factor  in  program  implementation 
was  addressed  in  seven  papers.  Social  issues  within  developing  countries  might  foster 
mistrust  between  staff,  the  program,  and  targeted  beneficiaries  (22;  106).  The  authors  of 
three  papers  addressed  the  social  atmosphere  in  which  the  program  operated  (27;  40;  50) 
and  how  staff  adopted  a  caring  ethos  that  permeated  not  only  a  direct  care  relationship, 
but  also  the  local  community  in  general  (50). 

The  availability  or  skill  level  of  human  resources  to  deliver  a  health-related 
program  was  addressed  in  ten  papers.  The  most  cited  challenge  to  human  resources  for 
programs  in  these  developing  countries  was  staff  absenteeism  (4)  and  turnover  (4;  13;  18; 
28;  37;  5 1 ).  Human  resource  capacity  at  the  national  and  sub-national  levels  was 
identified  as  an  important  constraint  by  the  authors  of  four  papers  (6;  27;  44;  50).  On  a 
micro  level,  the  presence  of  worker  mistrust,  attitude  of  helplessness  and  lack  of 
motivation  was  also  cited  as  personnel  issues  that  hindered  program  delivery  (28). 

Operating  a  new  program  using  existing  infrastructure  and  local  capacity  could 
lead  to  challenges  during  the  stages  of  both  health  program  design  and  implementation. 
The  authors  of  two  papers  addressed  the  importance  of  an  existing  health  care  structure  in 
program  implementation  (22;  40).  Depending  on  the  program,  a  basic  level  of 
infrastructure  was  critical,  including  such  elements  as  roads  and  transport  for  supplies; 
telephones;  electricity;  and  available  computers  (22;  46;  57). 
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The  authors  of  four  papers  specifically  addressed  the  role  of  donors  and  external 
support  in  program  implementation  (4;  6;  40;  57).  Key  attributes  of  program  support  that 
these  authors  identified  included  sufficient  and  flexible  financing  and  hands-on  support 
by  partner  agencies  (6). 

Attributes  related  specifically  to  disease  surveillance  systems 

The  authors  of  three  papers  described  their  personal  experiences  with  disease 
surveillance  programs.  They  identified  four  factors  important  to  surveillance  program 
design:  a)  simplicity  of  data  entry  (91 );  b)  simplicity  in  case  definitions  (syndromic 
approach )  (9 1 );  c)  training  in  data  analysis  (91);  and  d )  data  quality  through 
standardization  of  data  at  the  national  level  (91). 

Program  acceptability  was  unfavorably  affected  by  several  factors  including  a 
lack  of  connection  between  the  program  and  response  and  control  measures  (46;  9 1 ),  a 
perception  of  the  program  not  being  interoperable  within  a  routine  system  (91 ),  and  a 
lack  of  accountability  for  low  performance  (46;  91).  Increased  surveillance  program 
acceptability  at  a  national  level  was  associated  with  the  timely  reception  of  data  (91). 
Close  monitoring  was  considered  key  in  the  implementation  process  (69)  as  was  the 
flexibility  of  the  surveillance  system  in  adapting  to  feedback  from  surveillance 
stakeholders  (69). 

Among  the  engagement  factors  identified  by  the  researchers  in  this  review  were 
an  improvement  in  feedback  and  communication  at  all  levels  and  among  all  surveillance 
actors  (91 ),  broad-based  training  for  reporting  staff  (46),  and  a  commitment  of  resources 
to  afford  a  better  response  (69).  Similarly,  poor  dissemination  of  data,  and  lack  of 
feedback  to  surveillance  staff  on  their  performance  (69)  were  suspected  reasons  for  lack 
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of  staff  motivation.  Additional  environmental  factors  included  the  availability  of 
resources  at  the  surveillance  sites  such  as  phones  and  electricity  (46). 

In  summary,  a  systematic  review  of  the  research  literature  related  to  health 
program  implementation  revealed  that  program  design,  accountability  and  leadership, 
target  audience  and  stakeholder  engagement,  and  environmental  factors  were  important 
considerations  in  the  implementation  of  such  programs  in  developing  countries.  Training 
and  flexibility  were  that  emerged  as  program  design  issues.  Germane  to  the  area  of 
accountability  and  leadership  were  accountability  policies,  leadership,  supervision,  and 
the  need  to  address  monitoring  and  evaluation  processes  such  as  the  identification  of 
indicators  to  be  evaluated.  Key  considerations  in  engaging  target  audiences  were  issues 
of  communication,  incentives  use,  stakeholder  involvement,  and  local  capacity  building. 
Related  to  the  environment  in  which  the  health-related  program  was  implemented,  the 
political  and  social  environment  were  key  considerations  as  were  the  availability  and 
skills  of  human  resources,  existing  infrastructure  and  capacity,  and  the  role  of  donor 
financing  and  need  for  external  hands-on  sources  of  support  (See  Figure  1 ). 

Although  factors  identified  in  the  literature  review  might  provide  initial  guidance, 
it  was  evident  that  a  need  existed  for  research  on  factors  affecting  the  successful 
implementation  of  disease  surveillance  programs  in  developing  countries  specifically  (20; 
64).  Therefore,  the  researcher  developed  this  study  to  address  the  knowledge  gap  in  this 
area.  This  work  was  structured  around  three  aims:  a)  to  expand  the  literature  review  by 
searching  the  gray  literature  in  order  to  identify  implementation  factors  associated 
specifically  with  electronic  disease  surveillance  systems  in  developing  countries;  b)  to 
convene  experts  in  disease  surveillance  and  related  areas  in  an  effort  to  define  the 
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purpose,  operation,  and  the  factors  related  to  successful  implementation  of  electronic 
disease  surveillance  systems:  and  c)  to  develop  an  initial  framework  informed  by 
Grounded  Theory  methodology  based  on  the  collection  and  analysis  of  primary  research 
data  related  to  program  implementation  from  electronic  disease  surveillance  programs  in 
Peru. 

The  initial  framework  developed  in  aim  #3  was  revised  after  triangulating  data 
collected  in  the  previously  described  systematic  literature  review,  the  results  of  the  gray 
literature  search  (aim  #1),  and  the  conclusions  of  the  expert  consensus  workshop  (aim 
#2). 

The  revised  model  provides  an  understanding  of  all  of  the  components  of  a 
surveillance  system,  key  stakeholders,  and  activities  that  must  be  performed  and 
identifies  the  barriers  to  success  in  global  settings  that  will  enable  public  health  officials 
to  design  more  effective  disease  surveillance  systems.  Effective  disease  surveillance 
systems  will  better  meet  the  needs  of  developing  countries  and  the  several  governmental 
and  non-governmental  organizations  that  fund  these  initiatives  (e.g..  Department  of 
Defense  Global  Emerging  Infections  Surveillance  and  Response  System  (GEIS),  and  The 
Bill  &  Melinda  Gates  Foundation).  Such  surveillance  systems  will  lead  to  more  effective 
use  of  resources  (19),  improved  disease  surveillance,  and  ultimately  the  improved  health 
status  of  the  population. 
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CHAPTER  2:  MATERIALS  AND  METHODS 


As  addressed  in  the  previous  chapter,  this  study  was  structured  around  the 
development  of  three  aims.  Findings  from  a  systematic  literature  review  presented  in  the 
background  chapter  indicated  the  limited  literature  available  to  guide  the  implementation 
of  electronic  disease  surveillance  systems  in  developing  countries.  Therefore,  the  first 
aim  involved  expanding  the  research  literature  review  to  include  the  grey  literature  in 
order  to  identify  such  factors.  To  complement  this  effort,  the  researcher  organized  a 
workshop  that  convened  experts  in  the  field  of  electronic  disease  surveillance 
implementation  to  reach  consensus  on  surveillance  implementation  factors. 

Aims  #1  and  #2  provided  the  foundation  for  aim  #3,  ithe  outcome  of  which  was 
the  development  of  a  model  or  framework  for  the  successful  implementation  of  electronic 
disease  surveillance  systems  in  developing  countries  based  on  primary  data  collection  in 
Peru.  The  model  that  emerged  from  the  data  collected  in  Peru  was  then  triangulated  with 
data  from  three  additional  data  collection  strategies  used  in  this  research:  systematic 
literature  review,  grey  literature  review,  and  expert  consensus  workshop.  The 
triangulation  of  data  led  to  the  creation  of  a  new  refined  model  that  provided  a  more 
comprehensive  insight  to  implemented  of  such  systems  in  resource  constrained  settings. 

Aim  #1  -  To  expand  a  preliminary  literature  review  by  searching  in  the  gray 
literature  for  factors  associated  with  the  implementation  of  electronic  disease 
surveillance  systems  in  developing  countries. 
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The  first  aim  expanded  the  preliminary  systematic  review  of  the  literature  by 
including  gray  literature  sources.  The  gray  literature  refers  to  publications  that  are  not 
indexed  in  traditional  biomedical  and  nursing  databases  and  are  not  controlled  by 
commercial  publishing  interests.  It  is  a  gray  literature  that  is  comprised  of  documents 
issued  by  government,  academia,  and  business  and  industry,  such  as  newsletters,  reports, 
working  papers,  theses,  dissertations,  government  documents,  bulletins,  fact  sheets  and 
conference  proceedings  (95  ).  The  Internet  was  used  as  the  major  source  of  gray  literature 
in  this  topic. 

The  additional  gray  literature  search  was  necessary  to  identify  specific  documents 
that  described  successful  implementation  practices  for  electronic  disease  surveillance 
systems  in  developing  countries.  In  the  development  of  aim  #1,  the  researcher  initially 
searched  for  documents  on  Popline,  NTIS,  ERIC,  New  York  Academy  of  Medicine 
(NY AM),  Worldcat,  and  Google  Scholar.  The  search  was  conducted  in  January  2012  and 
periodically  repeated  during  a  one-year  period  from  January  to  December  2012  to  ensure 
the  capture  of  any  newly  available  documents.  No  starting  date  limits  were  used  in  this 
search  of  the  resources  mentioned  above. 

The  search  strategy  and  results  are  summarized  in  Figure  2.  A  different  search 
strategy  was  developed  for  each  of  the  resources.  The  investigator  used  a  broader  search 
strategy  for  ERIC,  POPLINE  and  NYAM  to  ensure  all  publications  related  to  the  topic 
were  included;  these  resources  revealed  very  limited  number  of  results  when  specific  key 
words  was  used  (“electronic  disease  surveillance").  On  the  other  hand,  when  using 
Google  Scholar  and  WorldCat,  the  investigator  applied  a  narrower  search  strategy  using 
the  terms  “electronic  disease  surveillance"  AND  “developing  countries"  because  of  the 
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large  volume  of  results  obtained  when  using  those  specific  key  works.  The  researcher 
sought  consultation  with  experts  on  disease  surveillance  from  the  Pan  American  Health 
Organization  (PAHO).  Recommendations  by  PAHO  led  to  retrieving  references  on  the 
World  Health  Organization  website,  including  their  Integrated  Disease  Surveillance 
bulletin,  the  Weekly  Epidemiological  Record,  and  the  African  Regional  Office.  Lastly, 
the  investigator  reviewed  references  cited  in  all  of  the  publications  selected  and  applied 
the  same  inclusion  criteria  described  to  those  references.  Documents  retrieved  were 
included  in  the  review  if  they  met  the  following  criteria:  a)  the  surveillance  system 
described  was  electronic;  b)  the  surveillance  system  functioned  in  a  “developing  country'1 
or  a  “developing  setting”  as  defined  by  the  International  Monetary  Fund's  World 
Economic  Outlook  Report,  October  2009;  and  c)  the  source  described  surveillance 
system  inputs,  activities,  outputs  or  outcomes,  or  an  evaluation  of  the  implementation 
process. 

Once  all  the  publications  were  selected  using  the  inclusion  criteria,  key 
dimensions  of  the  system's  implementation  were  documented  qualitatively.  Basic 
qualitative  coding  methodology  was  applied  to  the  documented  dimensions  in  an  effort  to 
identify  key  themes  used  in  reporting  the  findings.  The  researcher  identified  themes  after 
completion  of  the  review,  and  a  table  was  developed  to  summarize  the  findings  in  each 
publication  included  in  this  review.  As  presented  in  Appendix  J.  the  format  of  the  table 
included:  1 )  reference,  2)  country/extemal  support,  3)  details  of  system  and  tool,  4)  study 
design  and  analysis,  5)  factors  observed  to  contribute  (to  successful  implementation),  and 
6)  challenges  and  needs. 
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Figure  2:  Strategy  for  the  inclusion  of  literature  for  electronic  disease  surveillance  implementation  on  developing  countries. 
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Aim  #2  -  To  convene  experts  in  disease  surveillance  and  related  areas  in  an  effort  to 
define  the  purpose,  operation,  and  the  factors  related  to  successful  implementation 
of  electronic  disease  surveillance  systems. 

Aim  #2  was  developed  to  complement  and  compare  information  acquired  through 
the  previous  literature  reviews  performed.  In  conjunction  with  these  two  methodologies, 
the  expert  consensus  workshop  was  intended  to  provide  the  foundation  for  the  primary 
data  collection  in  Peru.  Data  from  Peru  provided  an  initial  model  to  understand  the 
implementation  process  of  electronic  disease  surveillance  systems  in  a  low  resource 
country.  Using  the  findings  from  the  expert  workshop  consensus  process  and  the 
literature  review',  the  researcher  developed  the  interview  tool  for  aim  #3.  Furthermore,  the 
expert  consensus  workshop  findings  and  conclusions  served  as  one  of  the  information 
sources  used  to  triangulate  and  refine  the  model. 

On  February  22,  2012,  the  researcher  held  an  expert  consensus  workshop  titled, 
“Electronic  Disease  Surveillance  Systems  Implementation  in  Developing  Countries," 
with  the  support  of  the  Armed  Forces  Health  Surveillance  Center  (AFHSC).  The 
workshop  hosted  a  panel  of  15  subject  matter  experts  in  the  field  of  electronic  disease 
surveillance  systems  who  discussed  factors  related  to  the  successful  implementation  of 
such  systems  in  resource-limited  settings.  Workshop  attendees  were  invited  based  on 
their  expertise  in  the  area  of  disease  surveillance  implementation  and  public  health 
informatics,  and  their  experience  in  resource  limited  settings.  Due  to  budget  constraints, 
the  participant  selection  was  focused  on  experts  working  in  the  Washington,  DC  area.  A 
list  of  the  participants  and  their  positions  at  the  time  of  the  workshop  is  presented  as 
Appendix  G. 
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The  planning  phase  for  this  workshop  involved  members  of  the  researcher’s  thesis 
advisory  committee  with  expertise  on  this  field.  Each  member  provided  a  list  of  potential 
participants  for  the  workshop  and  the  final  list  of  participants  to  be  invited  was  agreed 
upon  through  email.  The  researcher  held  a  meeting  with  her  committee  members  to 
develop  the  agenda  of  the  meeting,  and  identified  the  need  for  facilitators.  Several 
committee  members  volunteered  to  be  the  facilitators  for  the  discussion  sessions.  Two 
participants  agreed  to  be  note  takers  for  the  discussion  sessions.  The  logistics  for  this 
meeting  was  supported  by  the  AFHSC. 

Workshop  attendees  addressed  the  lack  of  literature  (published  and  gray)  in  this 
topical  area  and  a  need  for  guidance  on  improving  surveillance  capabilities.  The 
questions  of  primary  focus  included:  a)  what  is  the  definition  of  electronic  disease 
surveillance?  In  order  for  a  system  to  be  considered  electronic,  how  does  technology  need 
to  be  included  in  the  process  of  data  collection,  analysis  and  feedback?;  b)  Are  there 
stages  or  phases  in  the  process  of  implementation  of  an  electronic  disease  surveillance 
system  in  a  developing  country?  If  so,  describe  each  implementation  stage  or  phase, 
successful  implementation  milestones  for  each  stage  or  phase,  and  identify  the 
performance  attributes  to  be  prioritized  and  the  indicators  to  monitor  such  attributes;  c) 
What  are  the  key  factors  (e.g.,  technical  considerations,  surveillance  system  structure, 
core  and  support  surveillance  functions,  surveillance  quality,  financial  considerations, 
political  considerations,  ethical  and  cultural  considerations,  organizational  and  workforce 
determinants)  related  to  a  successful  implementation  of  an  electronic  surveillance  system 
in  a  developing  country? 
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The  desired  outcome  for  this  expert  workshop  was  the  development  of  consensus 
definitions  or  descriptions  of  concepts  relevant  to  electronic  disease  surveillance  purpose, 
operation,  performance,  and  implementation  in  developing  countries.  In  order  to  provide 
context  for  the  conversation,  the  researcher  prepared  a  comprehensive  review  of  the  gray 
literature  distributed  prior  to  the  in-person  meeting  held  in  Washington,  D.C.  The 
researcher  led  the  one-day  workshop,  which  was  initiated  by  a  presentation  by  one  of  the 
participants  on  defining  a  public  health  surveillance  system  and  its  components.  The 
researcher  then  presented  a  synthesis  of  the  systematic  review  of  literature  previously 
conducted  that  addressed  the  implementation  experience  of  developing  countries 
regarding  disease  surveillance  systems. 

Following  the  initial  presentations,  the  researcher  charged  the  workshop  attendees 
with  reaching  a  common  vision  of  electronic  disease  surveillance  systems  and  agreeing 
on  a  definition  of  electronic  disease  surveillance  systems.  Participants  were  assigned  to 
one  of  two  groups  tasked  with  responding  to  one  of  the  two  following  questions:  a)  what 
are  the  factors  related  to  a  successful  implementation  of  electronic  surveillance  systems 
in  resource-limited  settings?;  and  b)  What  is  the  process  for  planning  the  implementation 
of  such  systems? 

Two  sessions  of  two  hours  each  were  dedicated  to  each  of  the  group  discussions. 
After  every  session,  all  of  the  workshop  participants  re-convened  as  a  large  group  to 
present  their  conclusions  to  each  other,  discuss  the  conclusions,  and  ultimately  reach 
consensus  about  the  conclusions  they  had  discussed.  Every  session  had  a  facilitator  and  a 
note-taker.  The  facilitators  led  the  group  discussions  and  both  authored  a  final  summary 
of  the  sessions.  All  of  the  conversations  were  audio-recorded.  The  researcher  received  the 
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summaries  of  each  session,  and  affirmed  the  summarized  information  by  listening  to  the 
session  audio-recordings. 

The  researcher  developed  an  initial  document  summarizing  the  consensus  reached 
at  the  workshop  around  the  questions  described  earlier  and  emailed  the  draft  document  to 
all  participants  with  a  request  for  feedback.  After  incorporating  all  feedback  into  the 
initial  document,  she  e-mailed  the  final  revised  document  to  all  workshop  participants. 

Aim  #3  -  To  develop  an  initial  framework,  informed  by  Grounded  Theory 
methodology  and  using  primary  research  data,  related  to  the  implementation  of 
electronic  disease  surveillance  programs  in  Peru. 

The  need  to  focus  on  the  process  of  implementing  electronic  disease  surveillance 
systems,  described  in  the  Introduction  chapter,  warranted  a  qualitative  research  approach. 
The  researcher’s  plan  to  focus  on  the  processes  and  context  to  better  understand  the 
factors  related  to  successful  electronic  disease  surveillance  implementation,  and  the  need 
for  a  framework  to  explain  the  implementation  process  pointed  to  Grounded  Theory  as 
the  most  suitable  methodology  for  this  study. 

Grounded  Theory 

Grounded  theory  is  a  qualitative  research  methodology  that  was  developed  and 
first  described  by  Glaser  and  Strauss  in  1967.  The  methodology  involves  both  describing 
and  explaining  the  social  system  under  study,  and  consequently  generating  an 
explanatory  model  based  on  data  collected  from  the  social  system  (42:  65).  Grounded 
theory  utilizes  in-depth,  open-ended  interviews  as  a  primary  qualitative  data  collection 
strategy;  and  constant  comparison  in  the  analysis  of  data,  including  identifying  the  codes, 
concepts,  and  categories  that  emerge  from  the  data  based  on  participants’  direct 


21 


experiences  with,  in  this  case,  disease  surveillance.  The  ultimate  goal  of  Grounded 
Theory  is  to  inductively  build  a  model  or  theory  that  emerges  from  data  rather  than  from 
preexisting  hypotheses  or  theories  (33;  68).  The  underlying  assumption  of  grounded 
theory  is  that  people  construct  their  realities  from  the  symbols  around  them,  and  thus 
make  sense  of  and  order  their  social  world  (24;  42). 

Grounded  theory  uses  systematic  data  collection  and  analysis  procedures  that, 
when  used  appropriately,  contribute  to  a  credible,  rigorous  study.  This  methodology 
requires  theoretical  sensitivity  and  the  use  of  theoretical  sampling  (21;  65).  Theoretical 
sensitivity  relates  to  the  ability  of  the  researcher  to  gain  insight  into  (21 ;  65)  and 
understand  and  give  meaning  to  data  collected  in  order  to  detect  what  is  relevant  to  the 
research.  For  instance,  if  the  topic  upon  consideration  is  implementation  of  electronic 
disease  surveillance  systems  in  Peru,  having  theoretical  sensitivity  would  mean  the 
researcher  has  an  understanding  of  this  process  due,  for  example,  to  previous  professional 
experience  that  involved  leading  the  implementation  of  such  systems,  or  by  having 
reviewed  the  literature  on  this  topic.  The  researcher  has  gained  theoretical  sensitivity  as  a 
consequence  of  her  professional  experience  in  Peru  related  to  disease  surveillance. 

Theoretical  sampling  refers  to  the  selection  of  interviewees  based  on  the 
researcher's  ongoing  analysis  of  the  qualitative  data  the  researcher  has  collected  (21;  65; 
68).  In  this  study,  data  analysis  began  with  the  first  interview  and  continued  through  the 
course  of  data  collection.  As  the  researcher  gained  insight  into  the  implementation  of 
electronic  disease  surveillance  systems  in  Peru,  she  based  further  decisions  about 
participants  to  be  interviewed  on  her  ideas  and  theories  that  emerged  from  the  data  she 
had  collected  and  analyzed.  In  other  words,  she  selected  participants  in  an  effort  to  assist 
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her  in  clarifying  or  confirming  emergent  research  findings.  For  instance,  if  during  an 
interview  the  researcher  learned  that  a  system  was  implemented  through  a  partnership 
with  another  organization,  she  planned  to  sample  potential  interview  participants  from 
that  organization.  This  type  of  sampling  methodology  also  meant  that  the  researcher 
included  more  participants  at  the  reporting  staff  level  to  confirm  relationships  that 
emerged  from  the  interviews.  She  learned,  for  example,  that  the  surveillance  staffs 
motivation  was  increased  by  their  observation  that  important  decisions  were  made  in  their 
organization  based  on  the  data  they  had  collected  and  reported. 

Peru  as  an  ideal  country  to  pursue  this  research 

The  researcher  selected  Peru  for  this  study  based  in  large  part  on  the  country’s 
rich  source  of  data  regarding  disease  surveillance  systems.  Having  worked  in  this  country 
in  disease  surveillance,  the  researcher  had  previous  knowledge  of  electronic  disease 
surveillance  systems  functioning  within  different  institutions  (MOH  and  Armed  Forces) 
with  both  successful  and  unsuccessful  experiences.  Furthermore,  the  researcher  had 
access  to  potential  gatekeepers  that  made  it  possible  to  obtain  the  required  administrative 
support  to  conduct  this  work.  The  researcher  also  was  a  native  Spanish  speaker,  which 
allowed  her  to  conduct  the  interviews  in  the  participants’  language. 

Data  collection 

Data  collection  related  to  aim  #3  was  approved  by  the  USUHS  Human  Research 
Ethics  Committee.  In  addition,  the  investigator  obtained  letters  of  support  front  the 
Peruvian  MOH  and  Armed  Forces  in  an  effort  to  facilitate  the  data  collection  process. 
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The  inclusion  criteria  for  this  study  included  male  and  female  stakeholders  in  an 
electronic  disease  surveillance  system  at  four  levels:  the  institutional  administrative  level, 
the  technical  assistance  level,  the  national  and  regional  disease  surveillance  team  level,  and 
the  reporting  staff  level  in  Peru  who  voluntarily  consented  to  participate.  Stakeholders  were 
2 1  years  or  older  and  worked  or  had  worked  at  one  time  on  an  electronic  disease 
surveillance  system  that  was,  at  the  time  of  the  interviews,  functioning  for  at  least  one  year 
and  that  collected  information  on  human  infectious  diseases  for  which  notification  was 
mandated  in  Peru.  Exclusion  criteria  for  the  study  included  stakeholders  associated  with 
electronic  disease  surveillance  systems  that  did  not  collect  information  on  their  monitoring 
and  evaluation  activities  related  to  disease  surveillance,  or  who  were  affiliated  with  systems 
that  had  been  functioning  less  than  one  year,  that  primarily  collected  data  on  non-notifiable 
diseases  or  that  was  set  up  for  research  purposes  only. 

Description  of  surveillance  systems  studied 

The  researcher  focused  data  collection  on  three  ongoing  electronic  disease 
surveillance  systems  in  Peru  that  had  functioned  for  at  least  one  year  prior  to  data  collection 
and  that  collected  information  on  monitoring  and  evaluation  activities  related  to  disease 
surveillance.  The  researcher  collected  data  on  each  of  three  systems,  one  in  the  Peruvian 
Navy  (Alerta  D1SAMAR),  one  in  the  Peruvian  Army  ( Vigila  COSALE),  and  one  in  the 
Ministry  of  Helath  (MOH  Sentinel  Surveillance  System).  These  three  systems  were 
purposively  selected  to  capture  similarities  and  differences  in  implementing  national 
disease  surveillance  systems  in  military  and  civilian  environments.  These  national 
institutions  had  a  need  to  implement  such  systems.  Furthermore,  the  surveillance  systems 
analyzed  in  the  military  were  the  only  systems  in  place  in  each  of  their  organization 
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(Peruvian  Army  and  Navy),  while  the  MOH  was  a  sentinel  surveillance  and 
complementary  system.  The  three  systems  had  units  that  reflected  high  performing 
surveillance  activities  and  others  with  low  performance  reporting  rates.  The 
characteristics  of  these  systems  provided  an  ideal  setting  for  conducting  this  study;  each 
system  will  be  described  below.  At  the  time  of  this  study,  Alerta  DISAMAR  was  an 
innovative  electronic,  Internet  and  phone  based  system  used  by  the  Peruvian  Navy.  The 
Navy  used  a  comprehensive  disease  surveillance  tool  for  mandatory  disease  notification. 
Personnel  associated  with  the  Alerta  DISAMAR  system  collected  and  analyzed  daily  and 
weekly  disease  reports  for  forty-five  diseases  and  syndromes,  including  mandatory 
diagnoses  required  by  the  Peruvian  MOH  and  disease  conditions  of  military  relevance. 
The  surveillance  process  involved  surveillance  personnel  in  initially  recording  individual 
and  grouped  disease  cases  on  pre-coded  forms.  Staff  then  manually  prepared  summary 
tables  and  entered  the  data  into  the  Alerta  system.  Pilot  operations  began  late  in  2002, 
moved  into  a  phased-in  strategy  in  2004,  and  finally  proceeded  to  total  program 
implementation  in  2005.  At  the  time  of  this  study,  Alerta  DISAMAR  received  reports 
from  approximately  121  surveillance  units,  including  those  based  on  ships. 

The  Peruvian  Army  implemented  a  system  called  Vigila  COSALE  in  2005  using 
a  tool  similar  to  Alerta  DISAMAR.  The  Peruvian  Army  had  somewhat  more  extensive 
surveillance  coverage  of  the  country  compared  to  the  Navy  since  it  received  surveillance 
data  from  approximately  153  surveillance  sites. 

The  MOH  Influenza  Sentinel  Surveillance  System  was  an  Internet-based  sentinel 
surveillance  system  that  captured  weekly  information  that  was  recorded  daily  from  Iwo 
sources:  a)  outpatient  visits  made  by  individuals  who  were  clinically  diagnosed  with  Flu- 
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like  syndrome,  and  h)  hospitalized  patients  diagnosed  with  severe  acute  febrile 
respiratory  syndrome  or  severe  acute  respiratory  infection.  The  influenza  surveillance 
system  was  linked  to  laboratory  diagnoses.  The  system  was  implemented  in  2005  and 
received  information  at  its  highest  demand  from  52  surveillance  sites  across  the  country. 

An  electronic  research  toolkit  was  developed  to  facilitate  data  collection  and 
contribute  to  the  rigor  and  credibility  of  the  study.  The  toolkit  included  the  following  data 
collection  templates:  research  log;  initial/ongoing/final  observations  of  the  country  in  the 
context  of  disease  surveillance  and  interview  setting;  interview  introduction;  in-depth 
interview  guide;  interview  and  observation  field  notes;  review  of  documents  template; 
interview  summary;  and  exit  synthesis  notes.  The  tool  kit  also  included  a  document  that 
listed  all  of  the  factors  related  to  the  successful  implementation  of  electronic  disease 
surveillance  systems  identified  in  findings  from  the  review  of  the  literature  and  the 
consensus  workshop  titled,  “Electronic  Disease  Surveillance  Systems  Implementation  in 
Developing  Countries,  as  presented  in  Appendix  J  and  Table  1 . 

Participant  recruitment  process 

The  researcher  initially  used  an  email  message  to  contact  the  person  in  charge  of 
the  implementation  of  the  electronic  disease  surveillance  system  in  each  institution  or 
organization.  There  were  no  fliers  or  public  recruitment  for  the  study  other  than  email 
invitations.  If  the  participant  expressed  interest,  the  researcher  requested  an  in-person 
interview  of  approximately  60  minutes  duration.  If  the  potential  participant  did  not 
contact  the  researcher  after  the  first  email  message  was  forwarded,  the  researcher  sent  a 
follow-up  email  after  one  week  of  the  initial  email.  If  there  was  no  response  to  the  second 
email,  the  researcher  did  not  contact  that  person  again.  Twenty-percent  of  the  participants 
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contacted  did  not  respond  to  the  second  email.  The  lack  of  response  may  be  explained  by 
the  researcher  not  having  their  current  email  addresses:  the  limited  capacity  of  some 
email  accounts  which  prevented  the  recipient  to  receive  emails  once  the  inbox  is  full;  or 
these  individuals’  limited  Internet  access. 

The  researcher  initiated  the  study  participant  recruiting  process  using  purposive 
sampling.  A  purposive  sample  is  one  that  is  selected  based  on  the  richness  of  the 
information  the  cases  will  provide  to  illuminate  the  questions  under  study.  There  are 
several  strategies  for  purposive  sampling  such  as  snowball  sampling,  which  identifies 
cases  by  asking  well  situated-people  for  potential  participants.  These  new  cases  are  then 
asked  about  new  participants  that  can  be  recruited,  making  the  “ball'  bigger  as  new 
people  are  asked  to  join  the  study  (72).  The  researcher  developed  a  glossary  with  the 
definitions  of  the  sampling  methodologies  used,  presented  as  Appendix  M. 

The  researcher  initiated  participant  recruitment  for  this  study  by  contacting  the 
head  of  the  Central  Disease  Surveillance  hub  at  the  U.S.  Naval  Medical  Research  Unit-6 
Peru  (NAMRU-6).  The  NAMRU-6  hub  provided  technical  and  financial  support  to  the 
electronic  disease  surveillance  systems  that  were  implemented  in  the  Peruvian  Army, 
Navy,  and  Air  Force,  as  well  as  other  countries  in  Latin  America.  In  NAMRU-6.  a  person 
from  each  Peruvian  military  organization  oversaw  the  performance  of  their  system  and 
worked  in  collaboration  with  NAMRU  staff. 

The  researcher  identified  a  Peruvian  Armed  Forces  representative  who  had 
significant  experience  with  the  Peruvian  disease  surveillance  system.  The  researcher  then 
used  snowball  sampling  by  asking  this  individual  and  other  subsequent  interviewees  to 
identify  leaders  and  others  affiliated  with  the  Armed  Forces  who  had  direct  experience 
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with  disease  surveillance  in  Peru  that  she  might  contact.  The  researcher  also  used 
criterion  sampling  to  select  individuals  to  interview  who  were  affiliated  with  the  MOH 
and  the  Armed  Forces. 

Throughout  this  study  the  researcher  used  the  qualitative  sampling  process  of 
purposive  sampling  that  involved  obtaining  insights  from  individuals  with  direct 
experience  with  disease  surveillance  in  the  country.  Criterion  sampling  served  as  one 
form  of  purposive  sampling  that  led  to  the  initial  inclusion  of  individuals  selected 
according  to  criterion  developed  and  articulated  by  the  researcher.  She  also  used 
snowball  sampling,  another  form  of  purposive  sampling. 

Consistent  with  the  qualitative  research  process  (72),  after  conducting  and 
transcribing  the  First  interview,  the  researcher  initiated  data  analysis.  At  that  point,  she 
began  to  use  theoretical  sampling,  which  is  a  purposive  sampling  process  associated  with 
grounded  theory  that  involves  the  selection  of  interviewees  in  response  to  the  researcher’s 
evolving  data-based  thinking  about  an  emerging  model  of  electronic  disease  surveillance 
implementation.  Intensity  sampling  was  used  to  identify  participants  who  had  important 
perspectives  and  rich  experiences  with  the  evolving  concepts  identified  by  the 
researcher.  A  commonly  shared  concept  of  externally  provided  technical  assistance 
emerged,  for  example.  The  researcher  then  interviewed  an  individual  with  extensive 
experience  providing  technical  assistance  during  the  process  of  implementing  several 
disease  surveillance  systems.  This  interview  enabled  the  researcher  to  gain  additional 
depth  and  insight  into  this  particular  aspect  of  surveillance  implementation. 

At  another  point  in  the  data  analysis  process,  the  researcher  had  gained  insight 
into  many  elements  of  the  surveillance  implementation  process,  but  recognized  a  need  to 
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explore  these  concepts  from  the  perspectives  of  different  stakeholders'  who  held  different 
roles  in  the  surveillance  process.  She  thus  began  identifying  and  interviewing  individuals 
at  various  levels  of  surveillance  responsibility  from  the  MOH  and  Armed  forces. 

Data  analysis  revealed  that  technical  issues  related  to  surveillance  tool 
characteristics  needed  to  be  further  explored.  She  then  pursued  interviews  with  engineers 
involved  in  developing  disease  surveillance  tools  in  Peru. 

The  researcher  also  recognized  that  in  order  to  better  understand  the  concept  of 
organizational  commitment  and  leadership;  she  proposed  that  the  starting  point  for  the 
emerging  surveillance  model  was  individuals  involved  in  Peru’s  initial  disease 
surveillance  implementation  efforts.  She  consequently  identified  and  interviewed  these 
individuals. 

The  researcher  sought  to  fully  explore  both  the  Armed  Forces  and  MOH  systems 
and  sought  to  gain  insight  from  individuals  from  the  organizations  that  comprised  these 
institutions.  She  interviewed  members  of  the  General  Division  of  Epidemiology  (GDE) 
and  Peruvian  National  Institute  of  Health  (NIH),  for  example,  in  order  to  gain  this 

""'Ife. 

organization’s  insight  into  surveillance.  These  interviews  affirmed  the  importance  of 
several  initial  observations  based  on  the  data.  Surveillance  systems  began  with  a  need 
that  arose  from  the  organization,  personnel  who  were  committed  to  the  surveillance 
process,  played  a  critical  role  in  the  system,  a  need  existed  to  enhance  staff  motivation 
and  identify  incentives  and  surveillance  efforts  had  to  be  coordinated.  The  researcher 
realized  that  she  had  to  explore  the  concepts  of  “organizational  needs”  and  the 
importance  of  coordination  coming  from  the  Armed  Forces’  perspective  in  order  to 
ensure  saturation  of  concepts  such  as  this  was  achieved.  She  conducted  interviews  to  the 
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point  of  perceived  data  redundancy  and  affirmed  saturation  by  conducting  additional 
interviews. 

The  interviews  the  researcher  conducted  helped  her  to  differentiate  between  those 
surveillance  sites  with  good  performance  and  those  that  did  not  perform  well.  The 
performance  of  a  site  was  considered  “good"  if  the  interviewee  referred  to  it  as  such. 
When  the  researcher  asked  about  the  meaning  of  “good"  performance,  participants 
generally  agreed  that  it  was  a  site  with  a  surveillance  reporting  rate  consistently  higher 
than  80%.  Contributions  to  the  differences  between  good  and  poor  performing  sites  were 
staff  motivation,  staff  knowledge  and  skills,  flexibility  in  surveillance  processes,  support 
from  site  directors,  and  the  presence  of  teamwork  at  the  site. 

At  this  juncture,  the  researcher  considered  it  important  to  confirm  these  concepts 
with  surveillance  staff  members  at  sites  that  participants  perceived  to  be  good  and  bad. 
With  the  collaboration  of  the  national  surveillance  teams  from  each  institution,  she  was 
able  to  schedule  group  and  individual  interviews  that  involved  personnel  from  the  two 
kinds  of  sites.  Data  from  these  interviews  clarified  the  concept  of  “awareness  of  the 
significance  of  the  disease  surveillance  tasks”  for  surveillance  staff  as  well  as  other  health 
and  non-health  workers  at  the  surveillance  units.  Interview  data  also  confirmed  that  the 
researcher  had  reached  saturation  on  the  “importance  of  feedback”  concept  that  enabled 
the  researcher  to  develop  the  concept  that  closed  the  model:  the  “value  and  action" 
concept. 

Surveillance  staff  repeatedly  raised  concerns  about  differences  in  the  situations  of 
staff  assigned  to  surveillance  sites  outside  the  capital  of  Lima  compared  to  those  assigned 
to  sites  closer  to  the  capital  and  other  major  cities.  Therefore,  the  researcher  considered  it 


30 


important  to  interview  a  person  who  had  exclusively  worked  in  a  geographically  remote 
location  to  explore  site-specific  characteristics  that  had  been  identified  by  surveillance 
staff  who  experienced  working  at  both  large  city  and  remote  locations.  These  interviews 
were  conducted  during  the  latter  part  of  data  collection  and  were  used  to  confirm 
saturation  of  the  concepts  already  identified  at  the  surveillance  staff  level.  At  a  point 
toward  the  completion  of  data  collection  the  researcher  believed  that  the  concepts  that 
explained  the  model  were  fully  developed.  However,  she  confirmed  saturation  by 
conducting  additional  follow-up  interviews.  This  strategy  was  particularly  challenging  as 
study  participants  were  busy  professionals.  Several  attempts  to  reach  two  potential 
participants  were  unsuccessful.  The  researcher  was  able  to  contact  and  interview  two 
final  participants  within  the  guidelines  of  the  ethics  committee  and  was  able  to  confirm 
saturation.  Figure  3  presents  a  diagram  of  the  recruiting  process. 

Interviewing  process 

The  researcher  conducted  all  in-depth,  open-ended  interviews  in  person  in  Peru. 
Once  an  individual  agreed  to  participate  in  an  interview,  that  person  and  the  researcher 
arranged  a  time  and  place  to  meet.  Most  interviews  were  conducted  in  the  individual’s 
workplace  in  a  private  office  setting  to  ensure  confidentiality.  All  interviews  were  audio 
recorded  and  conducted  in  Spanish  as  the  native  language  of  both  study  participants  and 
the  researcher. 

Before  initiating  the  interview,  the  researcher  used  a  template  script  to  describe  the 
purpose  of  the  study.  She  then  showed  participants  the  letter  of  support  from  their 
institution  or  organization  regarding  the  research  study.  If  the  participant  agreed  to 
participate  in  the  study,  the  researcher  engaged  the  participant  in  an  informed  consent 
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process.  An  informed  consent  document  was  shared  with  the  participants  (Appendix  N ), 
which  they  signed  if  they  agreed  to  participate. 

The  in-depth,  open-ended  interviews  the  researcher  conducted  during  the  study 
were  audio-recorded  with  participants’  permission  and  transcribed  verbatim  by 
transcriptionists  in  Peru.  Participants  were  not  identifiable  to  transcriptionists  who 
received  recordings  labeled  with  the  number  of  the  interview  rather  than  the  name  of  the 
participant.  Transcripts  were  not  linked  with  participant  consent  forms  to  ensure 
anonymity. 

The  researcher  developed  an  interview  guide  for  each  interview  that  she  tailored 
to  the  role  of  the  interviewee  and  purpose  of  the  interview.  A  sample  interview  guide  is 
provided  in  Appendix  H.  She  also  took  handwritten  notes  during  each  interview.  She 
initiated  the  interviews  by  asking  participants  about:  a)  reasons  the  disease  surveillance 
system  was  implemented  by  the  organization  or  institution;  b)  factors  or  stakeholders  that 
helped  the  implementation  process;  c)  challenges  faced  during  the  implementation 
process;  d)  factors  that  enabled  the  surveillance  system  to  currently  function  in  their 
organization  or  institution;  and  e)  how  successful  they  considered  the  particular  disease 
surveillance  system  with  which  they  were  affiliated.  These  questions  were  used  to 
identify  initial  domains  of  information  that  were  transformed  into  concepts  as  data 
collection  continued. 

Before  the  end  of  each  interview,  the  researcher  summarized  the  interviewee’s 
responses  and  asked  them  for  any  additional  comments.  All  in-person  interviews  were 
conducted  over  a  three-month  period  from  February  2012  to  April  2012  in  Peni.  The 
researcher  conducted  future  follow-up  conversations  with  interviewees  by  telephone  if 
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the  interviewee  had  agreed  to  further  contact  by  the  researcher  at  the  time  that  the  in- 
person  interview  was  conducted. 

Data  analysis 

In  summary,  the  researcher  used  the  process  of  Grounded  Theory  to  generate 
concepts  that,  in  turn,  guided  sampling,  data  collection,  and  analysis  (33;  68).  She 
continued  to  contact  potential  participants  via  email  (21).  Data  analysis  began  soon  after 
the  researcher  completed  the  first  interview.  Transcribed  interview  data  were  entered  into 
a  qualitative  software  program,  NVivo-9,  for  the  purpose  of  data  management  and 
analysis.  The  researcher  repeatedly  reviewed  each  transcript  in  its  entirety  in  preparation 
for  the  analysis.  While  the  researcher  was  analyzing  the  data,  she  wrote  analytical  memos 
that  were  conceptually  labeled  (21).  She  wrote  such  memos  for  the  purposes  of  open  data 
exploration  and,  identifying  and  developing  the  properties  and  dimensions  of  concepts. 
Axial  coding  enabled  her  to  relate  concepts  to  each  other,  draw  diagrams  to  indicate  the 
relationship  between  the  concepts,  and  to  visualize  the  initial  model.  She  used  selective 
coding  to  integrate  the  developed  categories  and  identify  the  core  variable.  An  example 
of  the  coding  process  is  described  as  Appendix  1. 
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Figure  3:  Participant  recruiting  process. 


34 


She  differentiated  levels  of  concepts.  Higher-level  concepts,  also  called 
Categories  or  Themes,  encompassed  lower  level  concepts  in  an  effort  to  reduce  the  data 
and  better  characterize  the  concept  121 ).  Once  the  relationships  were  clearly  identified 
and  the  concepts  were  integrated,  the  researcher  outlined  the  theoretical  scheme  (21). 

Audit  trail 

The  researcher  maintained  an  audit  trail  that  served  to  document  the  study  and  the 
analytical  strategies  employed.  The  audit  trail  comprised  three  key  documents:  a) 
research  log  (dated,  timed  registry  of  all  activity  and  decision  points  that  occurred  during 
the  research  process);  b)  researcher’s  journal  for  reflection  and  reflexivity  (dated  diary  of 
the  researcher’s  reactions,  assumptions  and  biases  related  to  the  data  collected  and 
analyzed);  and  c)  analytical  memos  (previously  described).  The  researcher  wrote  these 
documents  and,  with  the  exception  of  the  researcher’s  journal,  added  them  to  N-Vivo. 

The  researcher's  journal  was  kept  as  a  separate  document,  apart  from  all  other 
documents. 

Data  triangulation  in  developing  a  revised  model 

The  model  developed  using  grounded  theory  provided  a  solid  foundation  to 
understand  the  process  of  implementing  of  electronic  disease  surveillance  systems  in 
Peru.  However,  in  order  to  verify  the  validity  of  this  model  and  create  a  revised  version 
of  the  model  that  might  be  generalizable  to  similar  settings  external  to  Peru,  the 
researcher  incorporated  data  triangulation  in  developing  a  revised  model.  The  resultant 
new  model  was  based  on  the  four  sources  of  information  associated  with  this  study:  the 
systematic  literature  review  for  health-related  program  implementation  in  developing 
countries,  the  gray  literature  review  for  electronic  disease  surveillance  program 
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implementation  in  developing  countries,  the  expert  consensus  workshop  on  electronic 
disease  surveillance  program  implementation  in  developing  countries,  and  the  initial 
model  created  using  Grounded  Theory. 

Triangulation  is  defined  as  “the  combination  of  sources,  investigators, 
methodological  approaches,  theoretical  perspectives,  or  analytical  methods  within  the 
same  study”  (90;  96).  Therefore,  different  types  of  triangulation  exist:  theoretical 
tri angulation,  investigator  triangulation,  data  triangulation,  and  methodological 
triangulation,  depending  on  the  choices  the  researcher  makes  regarding  the  elements  to  be 
used  in  the  study  (104).  Triangulation  strategies  serve  two  purposes:  confirmation  of  the 
accuracy  of  the  results  by  using  different  approaches  or  methodologies  and  assuring 
completeness  in  order  to  provide  a  more  comprehensive  understanding  of  the  problem 
under  study  (84). 

Having  used  different  methodologies  for  data  collection  in  this  study,  the 
researcher  used  methodological  triangulation  to  revise  the  initial  Peruvian  model  by 
confirming  the  validity  of  the  concepts  already  identified,  and  by  incorporating  new  or 
refined  concepts  and  relationships  identified  by  the  other  three  methodologies.  With  this 
process,  the  researcher  aimed  to  provide  a  complete  picture  of  the  implementation 
process  of  electronic  disease  surveillance  systems  in  developing  countries. 

The  first  step  in  this  process  was  to  create  tables  to  show  the  concepts  in  the 
Peruvian  model  that  were  identified  by  the  other  methodologies  used  in  this  study.  For 
the  concepts  in  the  model  not  identified  by  additional  methods,  the  researcher  analyzed 
and  provided  reasoning  for  the  finding.  Following  this  analysis,  the  researcher  identified 
the  elements  in  each  of  the  other  three  methodologies  that  were  not  present  in  the  model. 
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These  concepts  were  assessed  regarding  the  methodologies  that  identified  them,  and  were 
incorporated  in  the  new  model  if  at  least  two  methodologies  used  in  the  study  addressed 
their  importance.  In  the  case  of  a  concept  present  in  the  findings  of  only  one 
methodology,  the  researcher  assessed  supportive  evidence  for  the  concept  in  order  to 
justify  its  inclusion  to  the  model.  In  the  case  of  discrepancies  in  the  concepts  identified  by 
the  different  methodologies,  the  researcher  investigated  them,  and  made  a  note  describing 
the  different  findings  and  offering  an  explanation  for  their  inconsistency. 

Finally,  the  researcher  developed  recommendations  based  on  this  revised  model 
for  implementers  desiring  to  establish  electronic  disease  surveillance  systems  in 
developing  countries,  or  for  those  public  health  professionals  desiring  to  improve  and 
refine  their  existing  disease  surveillance  systems. 
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CHAPTER  3:  RESULTS  -  CONTRIBUTION  OF  GRAY 

LITERATURE 


The  initial  systematic  review  of  the  scientific  literature  regarding  implementation 
of  health-related  programs  in  developing  countries  was  conducted  due  to  a  dearth  of 
literature  in  the  area  of  electronic  disease  surveillance  systems  in  developing  countries. 
The  review  yielded  important  findings  that  were  useful  in  initiating  a  process  of 
understanding  the  implementation  of  electronic  disease  surveillance  systems.  However, 
due  to  characteristics  unique  to  electronic  disease  surveillance  systems  (e.g.,  the  use  of  an 
electronic  tool,  electronic  reporting  processes),  the  researcher  expanded  the  literature 
review  by  conducting  a  search  of  the  gray  literature  that  specifically  addressed  the 
implementation  of  electronic  disease  surveillance  systems  in  developing  countries. 

Using  the  search  strategy  discussed  in  the  methodology  section,  1 ,405 
publications  on  the  topic  were  identified  using  gray  literature  search  tools.  1,393  of  which 
were  excluded  due  to  their  lack  of  relevance  or  failure  to  meet  established  inclusion 
criteria,  leaving  1 2  publications.  One  duplicated  reference  identified  in  the  scientific 
literature  was  found.  The  researcher  added  three  publications  as  a  consequence  of 
consultation  with  experts  and  10  more  by  identifying  cited  publications  within  the 
literature  identified.  A  total  of  24  publications  were  thus  identified  for  inclusion  in  this 
review  using  gray  literature  website  resources,  consulting  with  experts  on  the  topical 
area,  and  reviewing  the  citations  included  in  the  publications  identified. 

Regarding  the  geographic  locations  referenced  in  the  24  publications  included  in 
the  gray  literature  review,  five  described  disease  surveillance  systems  in  two  or  more 
countries.  Seven  publications  referenced  systems  centered  in  Africa,  5  based  their 
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research  in  Asia,  4  in  South  or  Central  America  and  3  in  Europe.  The  primary  objective 
of  the  gray  literature  review  was  to  identify  factors  related  to  the  successful 
implementation  of  electronic  disease  surveillance  systems  in  developing  countries.  The 
researcher  grouped  the  factors  revealed  in  these  publications  according  to  the  five  World 
Health  Organization  (WHO)  components  of  a  disease  surveillance  system  (Figure  4). 

According  to  the  WHO,  the  components  of  disease  surveillance  systems  address: 
a)  priority  diseases  for  surveillance,  b)  system  structure,  c)  surveillance  core  functions,  d) 
surveillance  support  functions,  and  e)  surveillance  quality.  These  components  provided 
the  basis  for  the  indicators  included  as  part  of  the  WHO  guidelines  for  monitoring  and 
evaluation  (101)  that  have  been  used  to  improve  disease  surveillance  system  performance 

The  results  of  the  gray  literature  are  summarized  below  and  the  researcher 
developed  a  table  in  which  the  details  of  each  publication  selected  were  presented 
(Appendix  J). 
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Figure  4:  WHO  components  of  disease  surveillance  systems  (101). 
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System  structure 

Within  the  WHO  system  structure,  the  researcher  identified  the  following  nine 
elements  of  electronic  disease  surveillance  systems  that  were  mentioned  in  the  gray 
literature:  political  commitment,  advocacy,  legal  framework,  financial  commitment, 
surveillance  strategy,  surveillance  tool,  system  organization,  accountability,  and 
development  of  partnerships  and  networks.  Each  of  these  elements  will  be  discussed 
further  in  the  following  section. 

Political  commitment 

Authors  of  publications  found  while  searching  the  grey  literature  mentioned 
political  commitment  as  a  critical  component  in  the  successful  implementation  of  a 
disease  surveillance  system.  They  indicated  that  obtaining  the  support  of  all  levels  of 
government  was  of  primary  importance  in  system  implementation  (38),  especially  during 
the  early  implementation  phase  of  the  implementation  process  (81).  In  addressing  the 
implementation  of  the  Ugandan  disease  surveillance  experience,  Lukwago,  Nanyunja, 
Ndayimirije,  Wamala,  Malimbo,  et  al.,  for  example,  identified  a  need  to  institutionalize 
the  surveillance  system  in  the  organization's  agency  and  the  public  health  budget  (61). 
According  to  the  WHO  and  others,  a  high  level  of  commitment  from  the  leaders  of  the 
organizations  involved  in  the  implementation  process  can  be  translated  into  the 
development  of  a  health  plan  and  policy  that  includes  surveillance  and  epidemic 
preparedness  as  major  components  (30;  86;  99),  and  that  assures  resource  allocation  to 
surveillance  (30;  98;  99).  Moreover,  as  illustrated  in  the  Ugandan  surveillance  project, 
Likwago  et  al.  observed  that  strong  support  from  the  Minister  of  Health  allowed  all 
health  workers  to  be  sensitized  to  the  relevance  of  surveillance  work  (98). 
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Advocacy 

Five  publications  in  the  literature  review  referred  to  a  need  to  achieve  political 
commitment  for  surveillance  through  the  formal  sensitization  of  political  and  surveillance 
stakeholders  (93;  98).  Advocacy  for  a  surveillance  system  proved  useful  in  several  areas: 
stimulating  awareness  of  the  importance  of  surveillance  among  stakeholders  (61;  76), 
securing  required  resources  (93),  and  generally  improving  the  system  (61).  According  to 
a  WHO  publication,  the  use  of  functional  surveillance  models,  demonstrating  cost- 
effectiveness  and  providing  regular  updates  on  the  performance  of  the  surveillance 
system  (99).  However,  the  implementers  of  a  disease  surveillance  system  in  Sri  Lanka 
warned  others  to  consider  the  time  needed  to  obtain  this  support  during  the  planning 
phase  of  the  system  (8 1 ). 

Legal  framework 

The  legal  framework  for  surveillance  was  identified  as  a  priority  in  implementing 
and  strengthening  disease  surveillance  systems  (93;  98;  99).  The  legal  framework  was 
also  cited  by  the  WHO  as  key  to  the  effective  coordination,  integration,  and  sustainability 
of  the  systems  (99),  and  to  support  the  funding  required  to  initiate  and  sustain  the  system 
(30;  102).  The  authors  of  a  United  States  Agency  for  International  Development 
(USAID)  supported  project  in  Georgia  described  utilizing  surveillance  legislation  to 
address  the  legal  and  administrative  barriers  that  may  prevent  the  effective  functioning  of 
the  system  (76). 

Financial  commitment 

While  not  the  only  element  needed,  a  review  of  the  progress  and  challenges  of  a 
disease  surveillance  system  initiated  in  Uganda  revealed  that  financial  commitment  was 
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related  to  an  increase  in  the  performance  of  the  surveillance  system  (61).  Williams  et  al 
who  assessed  a  disease  surveillance  system  in  the  Republic  of  Armenia,  identified  the 
strength  of  a  systems  as  originating  from  the  separation  of  preventive  from  curative 
medicine,  which  meant  having  an  independent  status  and  budget  for  surveillance  ( 105). 
Lack  of  dedicated  funding  was  described  as  a  challenge  by  the  authors  of  publications 
describing  disease  surveillance  in  the  developing  countries  of  Georgia,  Ghana,  Lithuania, 
and  Tanzania  (30;  38;  76;  77).  Franco,  Setzer,  and  Banke  noted  that  the  absence  of  funds 
affected  surveillance  personnel  supervision  and  training  as  well  as  the  support  of  the 
system  in  general  (38). 

The  authors  of  xix  publications  mentioned  the  presence  of  strong  donor  support 
that  enabled  developing  countries  to  implement  their  surveillance  systems  (30;  32;  38; 

81;  98).  At  the  same  time,  the  authors  of  the  Georgia  immunization  surveillance  program 
explained  the  importance  of  donors’  demand  that  governments  financially  commit  to  the 
system  in  order  to  ensure  its  sustainability  after  donor  assistance  ceased  (76).  It  was 
notable  that  the  commitment  of  the  government  of  Georgia  to  finance  a  surveillance 
system  in  that  country  was  achieved  by  developing  an  advocacy  strategy  and  materials 
for  the  government  to  understand  why  the  system  was  a  priority  for  the  country  (76). 

Surveillance  strategy 

The  development  of  an  overall  strategic  vision  was  presented  as  an  important 
task  before  implementation.  One  publication  mentioned  that  without  such  a  vision, 
donors  could  drive  the  system  towards  their  own  interests  instead  of  following  the 
organization’s  interests  (76).  Therefore,  this  publication  noted  that  the  system  should 
reflect  the  high  priorities  of  the  organization  where  it  will  be  implemented  (76).  The 
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importance  that  the  plan  of  action  is  based  on  the  assessment  of  the  needs  of  the 
organization  was  cited  (98).  For  one  author,  this  plan  should  also  consider  the 
administrative  arrangements  and  the  local  contexts  in  each  country  when  designing  the 
surveillance  strategies.  The  example  provided  was  the  selection  of  a  district-focused 
approach  for  their  disease  surveillance  system  in  a  decentralized  country  (99).  Advanced 
planning  was  recommended  in  one  publication,  especially  when  expanding  sites  for  the 
system  (61). 

Information  in  the  gray  literature  reflected  a  need  to  move  toward  the  integration 
of  surveillance  activities  since,  as  the  WHO  observed,  vertical  systems  had  proven  to  be 
expensive  and  difficult  to  sustain  (99).  As  illustrated  by  Kant  and  Krishnan,  authors  of  a 
case  study  that  addressed  disease  surveillance-related  information  and  communication 
technology  in  India,  the  integration  of  disease  surveillance  in  a  country  could  be 
accomplished  through  the  establishment  of  surveillance  units  at  national,  regional  and 
district  levels  (55). 

An  additional  element  for  inclusion  in  any  surveillance  strategy  was  the  use  of 
collected  information  by  decision  makers  (38;  105).  Reporting  on  the  Sri  Lankan 
infectious  disease  surveillance  system,  Robertson,  Sawford,  Daniel,  Nelson,  and  Stephen 
underscored  the  importance  of  identifying  system  outputs  and  their  utility  early  in  the 
system  design  process  (81).  At  the  same  time,  disease  surveillance  reforms  in  Georgia 
revealed  that  implementers  should  develop  materials  to  help  health  care  workers 
understand  and  see  how  the  data  collected  would  be  used  (76). 

Surveillance  tool 
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In  concert  with  the  recommendations  of  Jefferson,  Dupuy,  Clauder,  Texier,  and 
Green,  the  reliability  and  availability  of  surveillance  tools  was  essential  for  the 
performance  of  the  disease  surveillance  system,  as  an  unreliable  tool  provided  poor  data 
and  the  unavailability  of  the  tool(s)  to  reporting  staff  hindered  their  surveillance  activities 
(52).  The  WHO  reported  that  surveillance  implementers  recommended  maximizing  the 
use  of  information  technology  for  surveillance  activities  (99).  Robertson  et  al.  described 
the  use  of  familiar  technologies  such  as  basic  cell  phones  in  Sri  Lanka  disease 
surveillance,  which  not  only  enabled  timely  data  collection  and  transmission,  but  also 
decreased  training  times  (81 ).  In  Uganda,  however,  surveillance  implementers 
encountered  the  challenge  of  a  lack  of  standardized  surveillance  tools  and  software, 
where  several  organizations  and  partners  used  different  tools  for  data  collection  and 
dissemination  (98).  In  addition  to  maximizing  the  use  of  familiar  and  basic  information 
technology,  standardizing  reliable  surveillance  instruments  and  software  within  the 
system,  and  ensuring  the  availability  of  instruments  to  staff,  Robertson  et  al. 
recommended  that  surveillance  systems  must  guarantee  the  appropriate  security  of  data 
that  they  collect  (81 ). 

Organizational  aspects 

In  two  publications,  implementers  of  surveillance  systems  recommended  creating 
an  adequate  organizational  structure  with  clear  roles  and  responsibilities  at  every  level  of 
the  system  (38;  76).  Reflecting  on  disease  surveillance  in  Tanzania  and  Ghana,  Franco  et. 
al  mentioned  the  possibility  of  failure  if  the  surveillance  strategy  did  not  take  into  account 
key  organizational  and  workplace  issues  and  constraints  (38).  The  implementing  team 
should  be  prepared  to  perform  continuous  assessment  of  the  system  and  identify  the 
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problems  in  order  to  support  the  reporting  staff  in  fulfilling  their  surveillance  tasks  (38). 
Due  to  limited  resources,  programs  need  to  implement  collaborative  team-work  at  every 
site  and  at  different  levels  of  the  system  (86).  Team  spirit  and  trust  have  grown  among 
stakeholders  with  more  programs  working  within  integrated  surveillance  systems  (98).  In 
terms  of  surveillance  staff  retention,  the  WHO  recommended  that  donors  be  allowed  and 
encouraged  to  provide  the  necessary  financial  incentives  to  retain  effective  workers,  even 
if  such  salary  supplements  would  cause  disparities  and  would  not  bear  sustainable  in  the 
long  run.  A  need  existed  to  change  the  attitudes  and  behavior  of  the  health  workers  . 
Behavioral  change  was  critical  in  the  application  of  training  skills  (99). 

Accountability 

Franco  et  al.  identified  the  need  for  accountability  as  part  of  an  overall 
surveillance  strategy  (38).  Both  Franco  et  al.  and  John,  Samuel,  Balraj,  and  John 
observed  that  the  absence  of  effective  enforcement  strategies  within  surveillance  systems, 
particularly  the  reporting  process,  led  to  a  low  performing  system  (38;  53).  However, 
implemented  also  commented  that  some  enforcement  efforts  could  be  counterproductive 
and  lead  to  a  negative  reaction  by  reporting  staff.  For  instance,  Franco  et  al.  reported  that 
administrators  in  a  Tanzanian  surveillance  system  attempted  to  hold  the  salaries  of 
reporting  staff  if  surveillance  reports  were  not  submitted  on  time.  This  enforcement 
strategy  was  unsuccessful  in  that  it  did  not  improve  performance  and  created  an  even 
greater  negative  perception  of  the  system  (38). 

Partnerships 

The  WHO  identified  the  establishment  of  partnerships  with  public  and  private 
agencies,  as  well  as  with  stakeholders  from  the  central  and  peripheral  levels  as  critical  to 
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the  successful  implementation  of  disease  surveillance  systems  (99).  The  WHO  and 
Franco  et  al.  noted  support  from  international  partners  in  implementing  disease 
surveillance  systems,  including  resource  mobilization,  formulation  of  training  policy,  and 
development  of  standards  and  guidelines  (38;  99).  For  instance,  Lithuania  took 
advantage  of  the  Swedish  experience  with  electronic  reporting  systems  to  support  the 
implementation  of  their  system  (30).  Technical  support  from  the  Centers  for  Disease 
Control  and  Prevention  (CDC)  (Atlanta,  Georgia)  and  WHO  was  used  to  improve  the 
performance  of  a  Ugandan  surveillance  system  (61).  However,  according  to  Robertson  et 
al.,  technical  assistance  should  focus  on  developing  local  expertise  from  the  start  of  the 
process  in  order  to  ensure  sustainability  of  the  system  (81).  Forging  partnerships  fostered 
the  strengthening  of  surveillance,  training,  and  laboratory  networks  at  the  local,  national, 
and  regional  levels  (56;  99).  For  these  networks  to  work  efficiently,  however,  the 
existence  of  a  strong  national  coordination  body  (56;  99)  and  effective  coordination 
mechanisms  proved  useful  (32;  99).  These  coordination  efforts  were  especially  needed  at 
the  regional  and  district  levels,  as  well  as  the  MOH  level  in  surveillance  systems  ( 102). 
Surveillance  core  functions 

Within  the  surveillance  core  functions,  the  information  revealed  by  the  gray 
literature  included  data  analysis  and  interpretation,  and  the  feedback  processes.  Each  of 
these  core  functions  will  be  addressed  in  the  following  section. 

Data  analysis 

Due  to  analysis  capacity  challenges  at  the  lower  levels  of  surveillance  systems, 
the  gray  literature  reviewed  suggested  using  epidemic  thresholds  based  on  technical 
guidelines  to  improve  outbreak  detection  (98;  99)  and  the  development  of  automated 
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analysis  to  facilitate  tasks  at  the  local  level  (87).  The  WHO  also  noted  a  need  for 
improved  data  management  at  the  central  level  (99). 

Feedback 

Feedback  was  deemed  an  important  component  of  the  system,  particularly  in 
efforts  made  to  inform  health-related  surveillance  workers  about  the  data  they  collected 
and  how  the  data  were  used.  Feedback  could  be  offered  through  weekly  newsletters, 
website  updates,  press  releases,  field  investigation  reports,  and  quarterly  bulletins  (99; 
102).  The  WHO  recommended  that  such  feedback  be  produced  regularly  and  in  a  timely 
way  at  the  central  level  (99).  In  an  evaluation  of  a  surveillance  system  in  South  Africa, 
Weber  recommended  the  dissemination  of  surveillance  results  to  a  wide  audience  of 
health  care  providers  with  a  clear  explanation  about  how  the  information  collected  had 
influenced  the  response  to  outbreaks  or  was  used  to  assess  the  public  health  interventions 
that  had  been  implemented  (94).  Understanding  the  importance  of  disease  surveillance 
information  served  to  motivate  reporting  staff.  Feedback  was  also  recognized  as  a  critical 
element  in  an  effective  accountability  system  (38).  John  et  al.  and  Wuhib,  Chorba, 
Davidiants,  MacKenzie,  and  McNabb  noted  that  a  failure  to  report  feedback  to  reporting 
staff  created  a  barrier  to  realizing  improvements  in  the  performance  of  these  staff 
(53;  105).  Weber,  for  instance,  noted  that  staff  who  received  no  feedback  shared  that  they 
did  not  perceive  any  public  health  use  of  the  data  they  had  collected  and  reported  (94) 

Surveillance  support  functions 

Within  surveillance  support  functions,  the  gray  literature  offered  information  on 
establishing  standards  and  guidelines,  training,  supervision,  communication,  resources, 
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and  monitoring  &  evaluation.  Each  of  these  support  functions  will  be  addressed  in  the 
following  section. 

Standards  and  guidelines 

The  implementation  of  a  disease  surveillance  system  requires  clear  policy 
directives  and  guidelines  for  all  surveillance  functions  (85).  These  guidelines  should  be 
standardized  at  the  central  level,  which  ensures  surveillance  is  conducted  using  similar 
methods  and  resources  at  all  surveillance  sites  throughout  the  country  (38).  These 
guidelines  should  include  rules  and  procedures  for  data  collection,  transmission,  analysis 
and  dissemination  (76).  In  addition,  a  standardized  protocol  for  identifying  and 
responding  to  outbreak  alerts  should  be  included  to  improve  timeliness  and  usefulness  of 
the  system  (52;  99).  These  documents  should  exist  and  be  available  in  formats  that  are 
easy  for  workers  to  understand  and  use  (38),  and  be  distributed  to  every  surveillance  site 
at  the  peripheral  level  (93;  102).  Kebede,  Gatabazi,  Rugimbanya,  Mukankwiro,  and 
Perry,  et  al.  described  how  the  use  of  multiple  forms  from  a  variety  of  partners  for  disease 
surveillance  duplicated  effort  and  wasted  resources  (56). 

An  additional  element  mentioned  in  the  area  of  standards  and  guidelines  was  the 
list  of  diseases  or  events  to  be  surveyed.  The  WHO  and  authors  of  surveillance  system 
assessments  in  Ethiopia  and  Uganda  recommended  that  national,  provincial  and  district 
health  departments  prioritize  and  update  their  lists  of  diseases,  considering  the  public 
health  importance  of  these  health  problems  (97-99).  In  addition,  standard  case  definitions 
for  priority  diseases  should  be  developed  and  distributed  to  all  health  facilities  (98;  105). 

Training 
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Training  of  health  care  personnel  was  considered  one  of  the  key  factors  in  the 
success  of  disease  surveillance  programs  (23;  87;  99;  105).  Training  was  key  in  system 
performance  (77)  and,  as  reported  by  Sow,  Alemu,  Nanyunja,  Duale.  Perry,  and 
Gaturuku,  contributed  to  the  improvement  of  timeliness  and  completeness  in  reporting 
surveillance  events  (88).  Weber  recognized  training  as  a  necessary  component  of  quality 
assurance  (94).  Moreover,  a  train  the  trainers  model  that  involved  training  health  district 
officers  in  training  and  supervising  new  staff  addressed  one  of  the  greatest  challenges  that 
these  countries  faced:  a  typically  high  staff  turnover  (80;  102).  A  key  training 
recommendation  revealed  in  the  gray  literature  was  to  implement  a  sustainable  strategy  to 
systematically  train  human  resources  in  surveillance  (85;  94;  99).  Robertson  et  al.  and 
Siswoyo,  Permana,  Larasati,  Farid,  Suryadi,  and  Sedyaningsih  emphasized  the  need  for 
ongoing  training  (81 ;  86),  including  developing  training  modules  as  part  of  a  routine 
program  of  continuous  education  (76;  77).  The  WHO  noted  a  lack  of  training  in 
epidemiology  across  all  health  care  professions  and  recommended  the  incorporation  of 
epidemiology  training  into  undergraduate  courses  (99). 

Kant  and  Krishnan  recommended  surveillance  skills  training  that  targeted  health 
workers  at  different  levels  of  the  system  (55),  in  order  to  provide  them  with  the  skills  to 
perform  the  tasks  required  (61),  especially  those  at  the  mid  and  peripheral  level  (99;  102). 
Due  to  the  different  skills  required  at  each  organizational  level,  courses  tailored  to  the 
target  population  of  learners  were  essential  (99).  Curioso,  Karras,  Campos,  Buendia, 
Homes,  and  Kimball  mentioned  the  need  to  customize  training  for  different  locations  as 
personnel  at  some  sites  needed  more  extensive  training  (23). 
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Regarding  the  content  of  training.  Sow  et  al.  recommended  that  skills  to  be 
addressed  include  core  surveillance  functions  such  as  detection,  data  analysis  and 
reporting  of  diseases  (88).  In  a  publication  recounting  the  implementation  of  an  electronic 
Peruvian  surveillance  system,  Soto  Araujo-Castillo,  Neyra,  Fernandez,  and  Leturia,  et  al. 
suggested  broadening  surveillance  training  to  include  topics  such  as  outbreak  response 
(87).  According  to  the  WHO,  effective  surveillance  training  included  the  provision  of 
appropriate  guidelines,  job  aids,  and  tools  (99). 

Supervision 

According  to  the  gray  literature  reviewed,  training  was  observed  to  be  critical  but 
insufficient  in  sustaining  the  implementation  of  a  disease  surveillance  system  over  time. 
Adequate  supervision  was  key  to  maintaining  and  improving  the  system’s  performance 
(38:  98).  More  specifically,  Sow  et  al.  pointed  to  supportive  supervision  at  the  district 
level  that  addressed  the  reporting  staff  s  surveillance  knowledge  as  likely  to  sustain  the 
performance  of  surveillance  (88).  Franco  et  al.  recognized  supervision  as  a  critical 
element  for  effective  accountability  (38).  Supervision  also  provided  an  opportunity  to 
train  staff  (38;  98)  and  address  problems  and  issues  at  the  site  (38).  According  to  Franco 
et  al.,  the  development  of  supervision  guides  was  needed  in  order  to  facilitate  this  activity 
and  ensure  that  capacity  building  would  occur  (38).  A  study  by  Huaman,  Araujo- 
Castillo,  Soto,  Neyra,  and  Quispe  JA,  et  al.  challenged  the  impact  of  supervision  on  data 
reporting.  These  authors  revealed  that,  although  monthly  supervisory  visits  associated 
with  a  surveillance  system  in  Peru  allowed  closer  contact  with  reporting  personnel  and 
the  consequent  improvement  of  data  quality  at  some  sites,  supervision  did  not  improve 
the  timely  submission  of  surveillance  data  (47). 
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Communication 


Communication  was  considered  by  Robertson  et  aJ.  to  be  instrumental  for 
successfully  implementing  a  disease  surveillance  system  (81).  The  enhancement  of 
regular  bidirectional  communication  among  different  stakeholder  groups  could  be 
increased  by  the  quality  and  frequency  of  the  communication  by  the  groups  (55;  94). 
Defined  communication  channels  between  various  levels  of  the  health  system  should 
exist  and  be  usable,  especially  at  the  district  and  facility  levels.  In  the  same  way  (38), 
explicit  linkages  with  relevant  MOH  divisions  were  quite  useful  (56).  The  WHO 
contended  that  communication  was  an  important  means  of  informing  reporting  staff 
about  performance  expectations,  including  providing  a  list  with  defined  tasks, 
competencies  and  responsibilities  (99).  Furthermore,  communication  could  be  used  to 
raise  awareness  of  health  workers  about  the  basic  principles  of  disease  surveillance  and  to 
develop  a  commitment  to  the  task  of  surveillance  (98).  This  was  achieved  by  regular  and 
coordinated  dissemination  of  information  to  the  public  and  health  workers  (99).  Some  of 
the  more  useful  sensitizing  activities  involved  direct  communication  with  health  workers 
with  information  that  was  relevant  to  their  setting  (102). 

Resources 

The  success  of  the  implementation  and  continuation  of  a  surveillance  program 
was  related  to  the  availability  of  adequate  resources  for  the  activities  these  processes 
entailed  (38;  85;  99).  Sufficient  resources  for  implementation  were  deemed  important, 
such  as  collection  forms,  computers,  and  phone  lines  (38;  76;  86).  However,  one  of  the 
most  frequently  described  challenges  reported  in  the  literature  was  a  weak  general 
infrastructure  and  shortage  of  equipment,  supplies  and  reagents  (56)  and  in  some  cases. 
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lack  of  resources  at  the  national  level  (30).  Therefore,  the  using  available  resources  and 
adapting  tools  and  strategies  already  in  place  was  a  strategy  strongly  recommended  in 
order  to  ensure  the  sustainability  of  the  system  (30;  38;  81;  87;  99). 

Regarding  the  technology  used,  Robertson  et  al.  reported  that  open-source 
software  options  were  preferred  over  proprietary  options  due  to  the  lack  of  or  reduced 
costs  of  open-source  options  (8 1 ).  An  expansion  of  a  surveillance  system  required  careful 
consideration  of  the  availability  of  resources,  not  only  surveillance  staff  for  each  new  site 
but  the  infrastructure  that  would  allow  the  staff  to  perform  effectively  (61).  One 
successful  story  described  by  Soto  et  al.  involved  adapting  a  surveillance  system  to  the 
existing  resources  of  the  organization  and  using  alternative  ways  to  transmit  the 
information,  such  as  the  use  of  radio  relay  to  expand  the  system  to  sites  where  no  other 
technology  was  available  (87).  The  use  of  radio  was  recognized  as  an  important  tool  to 
reach  remote  areas  with  poor  transportation  networks  and  lack  of  telephone 
communication  (102). 

The  WHO  and  others  identified  the  laboratory  capacity  at  surveillance  settings  as 
a  critical  challenge  that  included  the  limited  availability  of  qualified  personnel  and 
shortages  of  equipment,  reagents  and  staffing  at  the  district  level  (93;  98;  99).  The  WHO 
recognized  the  presence  of  a  laboratory  network  as  a  critical  component  of  surveillance 
systems  (99).  Therefore,  implementers  noted  the  need  to  strengthen  the  capacity  of  the 
laboratories  and  establish  referral  networks  within  private  and  public  sectors  (99).  The 
authors  who  described  a  successful  disease  surveillance  system  in  Peru  reported 
collaboration  among  different  laboratory  initiatives  in  the  region  instead  of  building  an 
entire  laboratory  network  in  an  effort  to  avoid  duplication  of  efforts  (87). 
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Human  resources 


The  largest  reported  barrier  to  successful  implementation  of  disease  surveillance 
systems  was  the  lack  of  adequate  human  resources  (32).  Several  publications  described 
the  challenges  surveillance  systems  faced  due  to  an  insufficient  number  of  skilled  human 
resources  (38;  56),  at  high  workload  for  a  limited  number  of  staff  (77;  94)  and  the  high 
level  of  attrition  (98)  in  these  settings.  Therefore,  the  need  to  develop  human  resources 
was  considered  critical  (32;  81;  86;  93;  98). 

Adequate  staffing  should  come  with  clearly  defined  roles  and  responsibilities. 
Moreover,  a  person  exclusively  dedicated  to  entering  surveillance  data  was  noted  to  be 
ideal  and  one  of  the  key  factors  for  the  success  of  a  system  in  Indonesia  described  by 
Siswoyo,  Permana,  Larasati,  Farid,  Suryadi,  and  Sedyaningsih  (86).  Conversely,  low 
levels  of  performance  in  a  Ugandan  system  were  related  to  not  having  a  dedicated 
surveillance  officer  at  the  provincial  unit  who  could  monitor  compliance  of  the 
surveillance  units  (94). 

While  adequate  numbers  of  skilled  personnel  were  critical  for  carrying  out 
surveillance  tasks,  high  performance  was  observed  in  sites  with  sustained  commitment  of 
the  health  facility  personnel  such  as  a  surveillance  program  in  the  African  region  reported 
by  Sow  et  al.  (88).  Siswoyo  et  al.  considered  the  commitment  of  the  reporting  staff  a  key 
component  to  the  success  of  the  implementation  and  the  sustainability  of  a  surveillance 
system  (86).  Robertson  mentioned  commitment  as  a  particularly  important  characteristic 
when  introducing  a  novel  surveillance  method  (81 ).  Domeika,  Kligys,  Ivanauskiene,  and 
Mereckiene  agreed  with  this  idea,  reporting  that  even  when  a  Lithuanian  staff  had  the 
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heaviest  workload,  they  were  motivated  to  change  to  an  electronic  reporting  system  due 
to  their  commitment  (30). 

For  commitment  to  be  achieved,  implementers  needed  to  convince  stakeholders 
that  their  surveillance  efforts  were  useful  and  their  valuable  time  was  not  being  wasted 
(87).  Several  strategies  to  motivate  reporting  by  staff  were  described  in  the  literature 
reviewed  (99).  These  included  creating  career  incentives  (97);  awarding  financial  (76)  or 
non-financial  prizes  to  districts  with  good  performance  on  the  basis  of  timeliness  and 
completeness  of  reporting  (98);  providing  medical  equipment  to  the  sites;  and  inviting 
reporting  staff  to  participate  in  annual  epidemiology  workshops  and  meetings  (80). 
Conversely,  there  was  also  a  mention  of  the  inability  of  a  system  to  develop  adequate 
financial  and  administrative  incentives  (76).  Wahib  et.  al  discussed  the  use  of  reported 
data  to  blame  and  sanction  reporting  staff  rather  than  to  implement  effective 
interventions.  This  behavior  hindered  outbreak  reporting  due  to  fear  of  these  sanctions 
(105). 

Monitoring  and  Evaluation 

The  WHO  contended  that  monitoring  was  critical  for  identifying  system  progress 
(99),  and  maintaining  and  improving  its  performance  (38).  Follow-up  was  also  noted  to 
be  an  important  component  of  the  accountability  system  (38).  Soto  et  al.  identified  having 
a  fully  dedicated  person  to  monitor  all  the  surveillance  sites  as  one  of  the  most  successful 
strategies  in  implementing  a  surveillance  system  in  Peru.  This  strategy  proved  to  be 
particularly  important  during  the  expansion  phase  of  the  system  (87).  A  study  of  this 
same  system  (Huaman  et  al.)  demonstrated  that  regular  phone  call  reminders  before  a 
reporting  deadline  improved  the  timeliness  of  reports  (47).  Franco  et  al.  reported  that 
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districts  in  Tanzania  and  Ghana  that  received  less  follow-up  did  not  improve  their 
performance  as  much  as  those  districts  that  received  follow-up  (38).  A  report  on  disease 
surveillance  in  Tanzania  revealed  that  national  and  regional  teams  were  instrumental  in 
ensuring  the  continuation  of  monitoring  activities  (77).  The  report  also  noted  that  it  was 
useful  to  provide  districts  with  tools  to  support  them  in  the  continuation  of  their 
surveillance  monitoring  activities  (77).  The  authors  of  three  publications  mentioned 
regular  evaluation  as  a  required  element  in  assuring  the  optimal  functioning  of  a  system 
(87;  94;  102). 

Surveillance  quality 

Within  the  surveillance  quality  component  of  disease  surveillance,  the  literature 
referred  to  two  elements:  simplicity ,  and  flexibility.  Each  element  will  be  addressed  in  the 
following  segment. 

Simplicity 

Five  publications  described  the  importance  of  having  a  simple  system.  Wuhib  et 
al.  identified  complex  systems  as  unsustainable  (105).  The  simplicity  of  the  surveillance 
task  was  related  to  the  acceptability  of  the  system  by  staff  (52).  Jefferson,  Dupuy, 
Clauder,  Texier,  and  Green  noted  the  importance  of  limiting  the  list  of  diseases  or 
syndromes  under  surveillance  or  the  amount  of  information  required  to  be  collected  in 
order  to  reduce  the  amount  of  time  taken  to  input  required  data  (52).  The  authors  of  two 
publications  commented  on  the  need  to  have  simplified  surveillance  forms  or  tools  (98) 
and  how  a  lack  of  surveillance  tools  negatively  impacted  surveillance  system 
performance  (88).  A  publication  authored  by  the  WHO,  CDC,  and  USAID  identified  the 
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need  to  lessen  the  burden  of  reporting  staff  by  minimizing  the  duplication  of  the  data 
collected  at  the  sites  (102). 

Flexibility 

The  WHO  mentioned  the  importance  of  the  surveillance  attribute  of  flexibility  in 
the  design  of  the  surveillance  strategy  in  order  to  address  the  surveillance  needs. 
Flexibility  was  noted  as  an  important  factor  when  designing  the  electronic  tool  used  for 
the  surveillance  processes  (99). 

In  summary,  as  shown  in  Figure  5,  elements  related  to  each  of  the  four  WHO 
components  of  disease  surveillance  systems  were  identified  as  important  to  the  successful 
implementation  of  electronic  disease  surveillance  systems  in  developing  countries.  The 
components  of  surveillance  structure  and  support  functions  offered  greater  opportunities 
for  implemented  to  increase  the  probability  of  succeeding  in  the  implementation  process, 
when  compared  to  the  other  two  components.  For  the  surveillance  structure  components, 
important  concepts  were  recognized,  including  polidcal  commitment,  advocacy  for 
surveillance,  legislative  framework,  financial  commitment,  surveillance  strategy, 
surveillance  tool,  system  organization,  accountability,  and  partnership  and  frameworks. 

In  the  support  functions  component,  the  importance  of  elements  such  as  standards  and 
guidelines,  training,  supervision,  communication,  resources,  and  monitoring  and 
evaluation  were  highlighted.  In  the  core  functions  component,  feedback  and  the  analysis 
of  the  data  concepts  were  considered  critical  to  enhance  the  performance  of  such  systems. 
Finally,  in  the  surveillance  quality  component,  two  attributes  of  simplicity  and  flexibility 
were  described  as  related  to  successful  systems.  Other  attributes,  measured  in  a  few 
publications,  related  to  surveillance  quality  included  completeness,  timeliness,  sensitivity 
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and  specificity.  However,  these  latter  attributes  were  not  described  in  the  literature  as 
critical  to  the  successful  implementation  of  an  electronic  disease  surveillance  system. 
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Figure  5:  Summary  of  literature  review  of  successful  implementation  of  electronic 
disease  surveillance  systems  in  developing  countries. 
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CHAPTER  4:  RESULTS  -  EXPERT  CONSENSUS  WORKSHOP  ON 
ELECTRONIC  DISEASE  SURVEILLANCE 

After  exploring  and  analyzing  the  literature  available  on  electronic  disease 
surveillance  program  implementation,  the  researcher  gained  consensus-based  insight 
from  experts  in  the  field  of  electronic  disease  surveillance.  She  convened  these  experts 
for  a  one-day  workshop  that  was  intended  to  address  the  three  following  questions 
outlined  in  the  Methods  section:  a)  what  is  the  definition  of  electronic  disease 
surveillance?  In  order  for  a  system  to  be  considered  electronic,  how  does  technology  need 
to  be  included  in  the  process  of  data  collection,  analysis  and  feedback?;  b)  Are  there 
stages  or  phases  in  the  process  of  implementation  of  an  electronic  disease  surveillance 
system  in  a  developing  country?  If  so,  describe  each  implementation  stage  or  phase, 
successful  implementation  milestones  for  each  stage  or  phase,  and  identify  the 
performance  attributes  to  be  prioritized  and  the  indicators  to  monitor  such  attributes;  c) 
What  are  the  key  factors  (e.g.,  technical  considerations,  surveillance  system  structure, 
core  and  support  surveillance  functions,  surveillance  quality,  financial  considerations, 
political  considerations,  ethical  and  cultural  considerations,  organizational  and  workforce 
determinants)  related  to  a  successful  implementation  of  an  electronic  surveillance  system 
in  a  developing  country? 

.  The  results  of  the  workshop  contributed  to  this  study  by  adding  experts'  in-depth 

'J, 

knowledge  on  electronic  disease  surveillance  in  general  as  well  as  surveillance  under  low 
resource  conditions,  informing  the  development  of  the  grounded  theory  study,  and 
providing  an  additional  key  source  of  data  that  ultimately  contributed  to  the  development 
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of  a  model  for  implementing  electronic  disease  surveillance  systems  in  low  resource 
countries. 

On  February  22,  2012,  Fifteen  subject  matter  experts  (Appendix  G)  in  electronic 
disease  surveillance  convened  at  the  Armed  Forces  Health  Surveillance  Center  (AFHSC) 
in  Silver  Spring,  Maryland  for  a  one-day  workshop  on  electronic  disease  surveillance 
implementation  in  developing  countries.  Workshop  outcomes  addressed  the  lack  of 
literature  (published  and  gray)  related  to  electronic  disease  surveillance  and  the  need  for 
guidance  on  improving  surveillance  capabilities  in  developing  countries.  The  desired 
outcome  for  this  expert  workshop  was  the  development  of  consensus  definitions  or 
descriptions  of  concepts  relevant  to  electronic  disease  surveillance  purpose,  operation, 
performance,  and  implementation  in  developing  countries.  The  agenda  of  the  workshop  is 
included  in  Appendix  K. 

Dr.  Patrick  Kelley,  National  Academies  of  Science  Institute  of  Medicine  Senior 
Director  of  the  Boards  of  Global  Health  and  African  Science  Academy  Development, 
provided  an  initial  overview  of  public  health  surveillance  systems  in  an  effort  to  set  the 
stage  for  the  workshop  discussion.  He  defined  disease  surveillance  in  the  following  way: 
A  system  for  public  health  surveillance  is  a  group  of  integrated  and  quality- 
assured,  cost-effective,  and  legally  and  professionally  acceptable  processes, 
designed  for  the  purpose  of  identifying  in  an  ongoing,  flexible,  standardized, 
timely,  simple,  sensitive,  and  predictive  manner  the  emergence  of  meaningful 
epidemiological  phenomena  and  their  specific  associations.  These  processes 
include  human,  laboratory,  and  informatics  activities  to  skillfully  manage 
information  derived  from  an  entire  defined  community  and  to  disseminate  that 
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information  in  a  timely  and  useful  manner  to  those  able  to  implement  appropriate 
public  health  interventions'  (49). 

After  significant  small  and  large  group  discussion,  workshop  attendees  reached 
consensus  regarding  a  definition  of  electronic  disease  surveillance  and  its  implementation 
in  developing  countries.  They  also  achieved  consensus  regarding  critical  elements  of  a 
disease  surveillance  system.  These  key  elements  included:  a)  clearly  understanding  the 
health  issues  or  events  that  required  surveying  before  developing  the  system;  b) 
identifying  what  was  to  be  surveyed,  including  whether  the  system  would  track 
infections,  syndromes  or  other  events;  c)  determining  the  data  collection  process  (face-to- 
face,  electronic,  or  a  combination  of  both);  d)  identifying  a  feasible  surveillance  question, 
considering  the  available  resources  from  the  country  and  donors;  e)  specifying  the  kinds 
of  data  to  be  collected;  and  f)  identifying  or  developing  a  data  collection  instrument  to 
address  the  purpose  of  the  surveillance  in  a  given  context.  Most  electronic  surveillance 
systems  selected  and  used  an  electronically-based  tool  to  facilitate  a  national  surveillance 
process  at  multiple  levels. 

Workshop  participants  identified  two  priority  areas  for  surveillance  system 
development:  a)  the  creation  of  global  system  architecture  for  a  surveillance  system;  and 
b)  the  development  of  human  resource  networks,  which  is  particularly  challenging  in  the 
developing  world  because  training  needs  existed  at  multiple  levels.  They  suggested  that 
surveillance  training  address  the  following  content  areas:  a)  how  to  plan  for  and  use  data 
that  were  collected,  b)  how  to  think  epidemiologically,  and  c)  how  to  analyze  data  and 
report  findings. 
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The  expert  participants  agreed  that  implementation  of  a  disease  surveillance 
system  required  consideration  of  inputs,  activities,  and  outputs  in  order  to  achieve  desired 
short  and  long-term  outcomes.  They  advocated  that  surveillance  professionals  adopt  a 
common,  longer-term  aim  of  effective  disease  control  rather  than  a  more  short-term  focus 
on  the  production  of  surveillance  tools  or  analytical  graphs.  Participants  concurred  that 
the  attributes  of  a  public  surveillance  system  described  by  the  Centers  for  Disease 
Control  and  Prevention  (CDC)  could  be  applied  in  the  framework  of  inputs,  processes 
and  outputs  (16). 

A  gap  in  surveillance  system  development  identified  by  workshop  participants 
during  their  discussions  was  the  absence  of  a  standardized  surveillance  implementation 
model  and  models  for  the  surveillance  processes.  During  the  discussions  the  experts  did 
not  reach  consensus  regarding  the  specificity  of  surveillance  system  development.  One 
expert  contended  that  surveillance  problems  arose  when  different  countries  and 
organizations  expended  large  amounts  of  resources  and  time  in  developing  customized 
solutions  to  their  surveillance  issues.  This  expert  suggested  a  need  to  develop  a 
standardized  basic  system  that  was  flexible  enough  to  be  adapted  to  different  realities. 
Alternatively,  other  experts  suggested  focusing  on  developing  situation-specific 
technology  to  fit  a  particular  situation  rather  than  developing  technology  to  meet  the 
general  needs  of  an  institution,  organization,  or  country.  While  consensus  was  not 
reached  on  this  issue,  the  need  for  system  flexibility  emerged  from  the  discussions 

Defining  electronic  disease  surveillance 

In  order  to  address  a  key  workshop  goal,  workshop  members  were  charged  with 
reaching  consensus  regarding  a  definition  of  electronic  disease  surveillance,  recognizing 
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the  increasing  use  of  electronic  tools  that  have  provided  new  and  innovative  ways  to 
conduct  surveillance.  The  experts  acknowledged  that  the  motives  for  and  outcomes  of 
disease  surveillance,  particularly  those  related  to  the  protection  of  public  health,  have 
changed  little  over  time.  The  methods  by  which  data  have  been  transmitted  and  the  tools 
available  for  transmission,  however,  have  evolved. 

The  experts  identified  two  main  areas  within  the  process  of  disease  surveillance 
where  electronic  tools  were  primarily  used:  data  collection  and  data  analysis.  They 
contended  that  a  system  could  be  deemed  electronic  if  either  one  of  those  processes  was 
electronic.  The  term  “electronic”  basically  meant  that  computer  software  was  used  in  the 
surveillance  process.  At  the  time  of  this  study,  the  ability  of  a  system  to  employ  an 
electronic  means  of  surveillance  was  an  important  consideration  for  surveillance 
developers  since  Internet  access  and  electronic  data  sources  were  rarely  consistently 
available  in  the  developing  world. 

Workshop  experts  acknowledged  that  almost  every  disease  surveillance  system  in 
developing  countries  required  staff  to  manually  enter  data  at  some  point.  Although  some 
data  could  be  directly  read  by  computer  software  such  as  laboratory  test  results,  other 
data  such  as  patient  demographics  and  diagnoses  were  typically  manually  entered  into  the 
system. 

Some  workshop  participants  suggested  a  change  in  terminology  that  involved 
replacing  the  term,  “electronic,”  with  a  reference  to  the  purpose  of  a  particular 
surveillance  system.  Surveillance  professionals  could  use  the  term,  “early  warning 
system.”  for  example,  which  may  or  may  not  be  electronic.  Members  of  the  workshop, 
however,  failed  to  achieve  consensus  on  this  point. 
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Implementation  phases  for  electronic  disease  surveillance  systems 

Workshop  participants  agreed  upon  five  key  phases  that  comprised  the  process  of 
electronic  disease  surveillance  system  implementation.  The  participants  considered  the 
context  of  implementation  as  one  in  which  an  external  agency  or  donor  organization 
supported  a  resource-limited  country  to  implement  a  surveillance  system.  The  phases  of 
implementation  identified  by  participants  included  partner  country  contact,  assessment  of 
preparedness,  development  of  a  plan,  system  implementation,  and  system  monitoring, 
each  of  which  will  be  discussed  below. 

Partner  country  contact 

The  first  phase  included  identifying  a  key  contact  person,  in  a  partner  country’s 

, 

surveillance  organization  who  had  the  capability  and  authority  to  support  the 
implementation  of  a  surveillance  system.  This  individual  might  be  the  Minister  of  Health 
or  the  Director  of  the  MOH  epidemiology  office  but  must  be  someone  who  could 
authorize  the  work  needed  to  plan  and  implement  a  surveillance  system  that  would 
ultimately  be  institutionalized  at  a  national  level 

Assessment  of  Preparedness 

Workshop  experts  identified  preparedness  assessment  as  the  second  phase  of 
surveillance  implementation.  Assessing  the  preparedness  of  the  institution,  organization, 
or  country  to  implement  an  electronic  surveillance  system  included  gaining  insight  into 
the  environment  in  which  implementers  would  be  working.  To  gauge  the  level  of 
preparedness,  implementers  must  determine  the  motivation  and  direction  of  the 
institution,  organization,  or  county  and  the  existence  of  or  potential  for  surveillance 
legislation. 
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Determine  motivation  and  direction. 


Workshop  experts  emphasized  the  importance  of  determining  the  type  of 
surveillance  system  in  which  a  particular  country  w-as  interested,  including  the  intended 
scope  of  surveillance,  particularly  whether  the  aim  was  to  survey  single  or  multiple 
diseases  or  conditions.  In  addition,  an  assessment  of  the  strength  of  political  will  was 
necessary  due  to  a  recognized  need  for  strong  political  support  in  the  implementation  of 
such  systems.  They  also  recommended  that  the  existence  of  political  motivations 
underlying  decisions  regarding  system  design  and  implementation  capabilities  be 
evaluated  and  the  expectations  of  country  leaders  be  assessed  for  evidence  of  realism  and 
rationality.  They  advised  system  implementers  to  address  the  country’s  expectations 
during  this  phase  of  the  process  in  an  effort  to  prevent  future  difficulties  and 
misunderstandings. 

Determine  the  existence  of  surveillance  legislation. 

Workshop  experts  agreed  that  identifying  government  policies  and  the  existence 
of  surveillance-specific  legislation  were  key  elements  to  be  addressed  in  this  phase  of  the 
process.  A  list  of  nationally  notifiable  diseases  was  one  of  the  first  documents  that 
workshop  experts  recommended  that  implementers  review. 

Assess  country  readiness. 

Workshop  experts  recommended  that  implementers  assess  the  readiness  of  the 
country  for  electronic  disease  surveillance  by  performing  a  basic  assessment  of  the 
infrastructure  for  surveillance  at  every  level  of  the  system.  This  infrastructure  included 
equipment,  phone  lines,  internet,  computers,  and  laboratory  capability.  In  addition,  they 
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contended  that  it  was  critical  to  assess  the  expertise,  knowledge,  and  ability  of  the 
surveillance  authority  and  surveillance  staff  at  every  level  of  the  system. 

In  order  to  develop  a  maximally  effective  and  efficient  surveillance  system, 
workshop  experts  directed  implementers  to  gain  insight  into  existing  surveillance 
processes  in  the  country  and  the  type  of  data  already  being  collected.  Such  insight  would 
enable  implementers  to  identify  collaboration  opportunities  and  effectively  integrate  new 
surveillance  needs  into  existing  surveillance  systems.  In  addition,  they  deemed  it 
essential  for  implementers  to  identify  all  surveillance  tools  in  current  use  and  their  user 
interfaces  in  order  to  develop  similar,  compatible  interfaces,  if  feasible. 

Assess  financial  support. 

Another  key  element  to  assess  in  the  process  of  developing  and  implementing  an 
electronic  disease  surveillance  system,  according  to  workshop  experts,  was  the  certainty 
of  the  funding  source  for  the  system  and  the  duration  of  the  sponsorship.  Additional 
donors/sponsors  should  be  considered  in  order  to  achieve  resource  diversification.  The 
experts  advised  implementers  to  convene  a  meeting  of  interested  donors  to  discuss 
surveillance  needs,  planned  activities,  and  resource  requirements.  Such  a  donor  meeting 
would  be  beneficial  in  determining  the  levels  of  commitment  (i.e.  financial,  technical) 
intended  by  each  potential  donor. 

Identify  sur\’eillance  needs. 

Experts  who  participated  in  the  workshop  underscored  the  need  for  implementers 
to  clearly  understand  disease  surveillance  priorities  and  needs  within  a  given  country.  For 
example,  implementers  must  identify  not  only  the  scope  of  surveillance,  including  the 
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range  of  diseases  to  be  surveyed,  but  also  the  personnel  required  and  levels  of 
engagement  by  and  commitment  of  health  care  system  personnel  who  are  involved  in 
reporting  data. 

Develop  a  Plan 

The  third  phase  identified  by  workshop  participants  as  the  development  of  an 
implementation  plan  that  is  based  on  the  assessment  described  above.  The  surveillance 
implementation  plan  should  be  based  on  a  strategic  plan  and  existing  surveillance 
processes: 

a)  Implemented  must  consult  the  strategic  plan  for  public  health  in  the  country  of 
interest.  Surveillance  represents  a  relatively  small  segment  of  a  national  public  health 
plan  and  should  be  designed  and  implemented  in  compliance  with  a  country’s  existing 
public  health  plan. 

b)  Use  surveillance  processes  already  in  place.  An  electronic  disease  surveillance 
system  should  be  designed  to  fit  the  processes  already  in  place  if  possible.  Implemented 
must  consider  the  ability  of  the  surveillance  system  to  communicate  with  other  systems  in 
the  future. 

The  workshop  experts  identified  planning  for  the  general  design  of  the  system,  a 
demonstration  project,  and  the  development  of  metrics  as  part  of  the  implementation 
phase  of  an  electronic  surveillance  system 

a)  In  terms  of  general  design,  implemented  should  plan  a  hierarchical,  top-down 
system  that  begins  at  the  national  office  and  progresses  to  the  most  distal  point  of  the 
system.  They  recommended  initiating  a  public  health  data  repository  at  the  national  level 


66 


for  data  collection,  analysis,  and  comparison  to  or  integration  with  other  existing 
surveillance  systems. 

b)  Demonstration  project  plans  are  advantageous.  Workshop  experts 
recommended  that  implementers  consider  conducting  a  demonstration  project  before  full- 
scale  implementation  of  a  system,  always  with  a  view  toward  larger  system  integration. 

c)  Metrics  should  be  developed  to  generate  the  information  needed  to  assess 
system  impact  and  processes  that  include  all  intermediate  phases  of  surveillance.  The 
experts  strongly  advised  considering  one  particularly  important  metric:  the  degree  to 
which  the  plan  adhered  to  ‘global’  standards  of  disease  surveillance  as  defined  by  the 
WHO  and  CDC(15;  100). 

System  implementation 

The  fourth  phase,  implementation  of  the  system,  involved  conducting  a 
demonstration  project,  scaling  up,  and  monitoring  and  evaluation,  each  of  which  is 
discussed  in  the  following  segment. 

a)  Demonstration  project.  The  system  implementation  aspect  of  the 
aforementioned  demonstration  project  consisted  of  using  a  means  of  collecting  feedback 
from  various  individuals  and  groups  to  determine  if  the  system  was  working  effectively 
from  both  an  informatics  and  end-users’  perspective.  The  experts  advised  implementers 
to  use  strategies  such  as  focus  group  interviews  to  obtain  feedback  crucial  in  refining 
surveillance  processes.  Not  only  did  insights  from  individuals  aid  in  designing  more 
efficient  and  effective  processes,  but  also  such  feedback  contributed  to  planning  relevant 
in-service  training  opportunities  for  surveillance  staff. 
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b)  Scale  up.  Workshop  experts  advocated  that  surveillance  sites  be  added  based 
on  available  resources  and  sufficient  monitoring  capacity  at  the  regional  or  national  level. 
During  this  period  of  implementation,  workshop  experts  stressed  the  importance  of 
implementing  a  continuous  rapid  cycle  assessment  that  fed  information  back  into  the 
system  for  the  purpose  of  improvement. 

c)  Monitoring  and  Evaluation.  The  experts  advised  that  strategies  in  the  areas  of 
monitoring  and  evaluation  be  established  prior  to  initiating  the  system.  The 
aforementioned  rapid  cycle  assessment  used  feedback  from  system  users  to  identify  ways 
to  improve  the  system  and  quickly  share  improvement  strategies  across  all  surveillance 
sites.  The  training  outcome  of  rapid  cycle  assessment  involved  in-service  education  that, 
in  some  cases,  included  bi-directional  training  in  that  users  taught  trainers  and  vice  versa. 
The  experts  believed  that  this  approach  also  supported  stakeholder  buy-in.  Workshop 
experts  recommended  that,  when  possible,  implementers  employ  automated  features 
developed  within  the  tool  to  ensure  the  quality  of  the  data  collected. 

System  Monitoring 

The  last  phase  of  implementing  an  electronic  disease  surveillance  system  as 
identified  by  the  workshop  experts  was  system  monitoring.  The  experts  based  their 
recommendations  for  system  monitoring  on  the  basic  program  logic  model  described  in 
the  CDC  guide  titled,  “Introduction  to  program  evaluation  for  public  health  programs”. 
The  elements  of  this  model  are:  Inputs,  Activities,  Outputs,  and  Outcomes  (16).  This 
model  was  used  to  cut  across  the  various  monitoring  areas. 

Timeliness  of  data  reported 
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At  the  national  or  regional  level,  timely  data  analysis  and  reporting  was  viewed  by 
the  experts  as  a  primary  activity  in  disease  surveillance.  In  order  to  assess  the  timeliness 
of  data  collection,  reporting  and  analysis,  they  recommended  that  the  surveillance  tool 
enable  the  recording  of  lag  time  between  disease  event  occurrence  dates  and  subsequent 
event  reported  dates.  Experts  contended  that  the  optimal  lag  time  between  the  clinical 
presentation  of  a  health  event  and  the  date  it  was  reported  (outputs)  was  no  more  than  72 
hours.  Workshop  experts  also  recommended  that  implementers  assess  the  difference  in 
time  between  the  data  collection  as  evidenced  by  form  completion  and  the  date  when  data 
were  entered  into  the  system. 

A  final  consideration  related  to  the  timeliness  of  data  reporting  was  system 
capacity  and  support  for  timely  decision-making.  According  to  workshop  experts,  one  of 
the  challenges  surveillance  system  implementers  have  traditionally  faced  has  been  a 
failure  to  initiate  public  health  interventions  at  a  time  when  data  suggested  that  such 
actions  might  reduce  morbidity  and  mortality.  Based  on  their  shared  experiences,  the 
experts  concluded  that  a  lack  of  response  to  data  findings  that  signaled  a  need  for  action 
was  generally  due  to  a  lack  of  analytical  capacity  on  the  part  of  surveillance  personnel. 

System  Flexibility/adaptability 

Workshop  experts  described  system  flexibility  needs  at  two  primary  levels:  the 
level  of  the  surveillance  instrument  (tool)  and  the  larger  system  level.  Implementers  must 
develop  surveillance  tool  features  that  enabled  respondents  to  address  new  questionnaire 
items  if  needed.  Implementers,  for  example,  might  be  requested  to  create  a  new  report  in 
response  to  International  Health  Regulations,  2005  (IHR  (2005)),  thus  necessitating  an 
change  in  the  tool  used  to  collect  data. 
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The  surveillance  system  itself  should  have  the  flexibility  to  incorporate  new 
directions  such  as  different  diseases  or  syndromes,  the  importation  of  different  data  sets, 
a  readjustment  in  the  categorization  of  data  or  a  change  in  data  analysis  (e.g.,  adding  new 
algorithms),  and  the  development  of  different  entry  profiles  and  data  presented  for 
different  jurisdictions.  One  advantage  of  a  flexible  surveillance  system  is  that  it  allows 
policy  makers  to  frame  reports  and  analyses  required  to  make  a  case  for  policy  changes 
that  ultimately  resulted  in  more  effective  disease  control. 

Stakeholder  acceptability 

Workshop  experts  insisted  that  any  surveillance  system  that  is  implemented  must 
be  legally  and  ethically  acceptable  to  stakeholders.  All  data  should  be  handled  in  a  legal, 
ethical,  and  professional  manner,  yet  be  legally  available  to  appropriate  surveillance 
personnel.  A  security  process  must  guarantee  the  protection  of  individuals’  privacy  by 
prohibiting  access  to  personally  identified  data.  Moreover,  protection  from  security 
system  violations  must  be  assured. 

An  additional  aspect  of  surveillance  that  workshop  experts  encouraged 
implementers  to  consider  in  their  efforts  to  develop  an  acceptable  surveillance  system 
was  the  sheer  volume  of  data  to  be  collected.  The  required  volume  of  data  should  be 
manageable  for  and  acceptable  to  surveillance  staff,  considering  the  resources  that  were 
available  to  them  and  the  workload  they  shouldered. 

Standardization 

Workshop  experts  advocated  the  standardization  of  surveillance  processes 
including  having  and  consistently  using  clear  guidelines  for  processes  and  employing 
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standard  field  case  definitions  and  methodologies  for  diagnosis  that  allowed  a 
comparison  of  outputs.  They  suggested  that  compliance  with  standards  supported  the 
generation  of  a  standardized  report.  They  also  reiterated  the  importance  of  assessing  the 
degree  to  which  a  system  complied  with  global  standards  that  would  allow  findings  from 
the  implemented’  surveillance  system  to  be  compared  to  findings  from  other  nations, 
thus  increasing  the  possibility  of  averting  transnational  public  health  emergencies. 

Usefulness 

The  usefulness  of  a  system  refers  to  the  ability  of  data  collected  by  a  particular 
surveillance  system  to  answer  the  questions  of  interest  posed  by  various  groups  of 
stakeholder.  In  addition,  the  users  of  the  system  should  be  able  to  group  the  data  as 
needed  to  learn  about  events  taking  place. 

User  friendly 

An  important  system-related  consideration  proffered  by  the  workshop  experts  was 
the  ease  of  system  use  by  surveillance  staff.  Modifications  to  the  surveillance  tool,  for 
example,  should  be  made  following  users’  feedback  and  based  on  staff  experience  using 
the  system  to  address  stakeholder’s  surveillance  questions. 

Quality/Validity 

Several  indicators  can  be  used  to  assess  the  quality  of  the  surveillance  data.  The 
workshop  experts  suggested  that  one  of  the  first  and  easiest  ways  was  to  monitor  each 
surveillance  site  for  the  proportion  of  records  found  to  be  complete.  For  any  surveillance 
system  to  be  considered  valid,  system  evaluators  needed  to  consider  the  sensitivity  and 
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specificity  of  data  needed  and  the  accuracy  and  completeness  of  reports  related  to  those 
needs. 

Scalability 

The  scalability  of  a  surveillance  system  was  defined  by  workshop  experts  as  the 
resources  required  by  implementers  to  use  a  developed  surveillance  tool  in  other  sites 
within  the  same  surveillance  system.  This  process  required  the  consideration  of 
limitations  and  needs  of  every  site  or  region  within  a  particular  nation  and  guides  the 
process  of  developing  or  selecting  and  implementing  a  surveillance  tool. 

Sustainability 

For  a  system  to  be  sustainable,  the  panel  of  experts  who  participated  in  the 
workshop  indicated  that  more  than  just  financial  resources  should  be  taken  into  account. 
In  their  view,  it  was  essential  that  governments  and  organizations  have  a  strategy  in  place 
to  develop  the  capacity  needed  to  sustain  the  system  when  donors  reduced  or  ceased  their 
support 

Portability 

Finally,  the  workshop  experts  contended  that  it  would  be  beneficial  for 
implementers  to  assess  the  portability  of  the  system.  Portability  was  particularly 
important  if  a  need  arose  to  translate  the  implementers’  experience  with  the  system  to 
other  localities  within  or  external  to  the  region  or  country  where  the  system  originated. 

Factors  related  to  the  successful  implementation  of  electronic  disease  surveillance 
systems  in  developing  countries 
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For  the  third  and  last  overarching  question  that  was  addressed  by  the  experts  who 
participated  in  the  workshop,  the  experts  agreed  that  system  Fit  and  sustainability  were 
factors  they  considered  most  clearly  related  to  the  successful  implementation  of 
electronic  disease  surveillance  systems  in  developing  countries. 

Participants  agreed  that  predicting  success  regarding  a  particular  surveillance 
system  depended  on:  a)  how  effectively  the  surveillance  system  Fit  the  needs  of  a  country 
or  organization,  and  b)  the  sustainability  of  the  system.  Workshop  participants  agreed  on 
ten  key  strategies  to  achieve  Fit  and  sustainability  (Table  1 ). 
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Table  1:  Factors  associated  with  successful  implementation  of  electronic  disease 
surveillance  systems  (Expert  consensus  workshop). 


Political  will 


Strategic  vision  and 
plan 

Adequate  finacial 
support 


•  At  every  level  of  the  system 
•Clear  articulation  of  needs 
•Legislation 

•Local  champion  (Regional  level) 

•To  map  out  sustainability 

•  Inlcude:  monioring  and  evaluation,  and  workforce 
development  plan 


•Buid  capacity  and  empower  surveillance  comunity 
•Avoid  creating  dependency  on  donors 


Trained  and  available 
workforce 

Performance 
improvement  plan 

Flexbility  and 
adaptability 

System  is  intuitive  to 
end-users 

Useful  at  all  levels 

Transparency  of  the 
data 

Start  small 


•Great  challenge  for  developing  countries 

•Provide  skills  to  produce  high  quality  data 

•Main  gaps  is  on  analysis  of  the  data  collected 

•Nurture  culture  to  keep  surveillance  staff 
motivated 

•Use  non-financial  incentives 

•  Provide  regular  feedback  to  end  users  of  the 
system 

•Personnel  and  software  need  to  adapt  to  changing 
environment 

•System  able  to  incorporent  different  and  new  data 
sources 

•Average  surveillance  staff  should  be  able  to  use  the 
system 

•Simple  but  if  needed,  more  complex  analysis  can 
be  performed 

•  From  local  users  to  the  national  level 

•  Feedback  is  important 


•  Develop  data  use  agreements 

•  Ensure  data  sharing  is  legal  and  ethical 


•System  should  be  tested  and  proved  to  work 
before  it  is  expanded  in  large  scale 
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Political  will 


According  to  workshop  experts,  political  resolve  about  the  need  for  and  value  of  a 
surveillance  system  had  to  be  present  at  every  level  of  the  system.  At  the  national  level, 
the  experts  believed  that  strong  executive  support  was  required  as  well  as  a  clear 
articulation  of  country  needs  by  the  authority  that  conducted  surveillance  in  the  country. 

The  highest-level  authority  was  advised  by  the  experts  to  develop  a  coherent  vision 
related  to  the  needs  of  the  country,  and  put  in  place  a  focal  point  or  central  coordinating 
office  to  oversee  the  system.  Political  will  needed  to  be  translated  into  regulation  or 
legislation  that  formalized  support  for  the  surveillance  system  as  well  as  the  commitment 
of  personnel  and  resources.  At  the  regional  level,  the  experts  contended  that  it  was 
important  to  have  a  local  champion  for  consistent  coherent  oversight.  The  implementers 
should  identify  a  person  who  believed  in  the  system  and  was  motivated  to  make  it  work. 
This  person  had  to  have  the  ability  to  engage  key  officials  and  influence  the  decision 
making  process.  According  to  the  experience  of  the  workshop  experts,  having  the  most 
senior  management  person  at  the  organization  involved  in  the  implementation  was 
correlated  with  success.  At  the  local  level,  it  is  important  to  work  closely  with  the  end 
users  to  ensure  usability  and  acceptability  of  the  system.  Ownership  of  the  system  needed 
to  be  present  at  all  levels. 

Strategic  vision  and  plan 

The  workshop  expens  deemed  the  presence  of  a  strategic  vision  and  plan  as 
critical  to  the  implementation  of  an  electronic  disease  surveillance  system.  They 
advocated  that  implementers  develop  a  strategic  plan  to  contribute  to  the  country’s 
ownership  of  the  system  and  to  map  out  strategies  for  sustainability.  For  this, 
implementers  needed  to  consider  the  sponsor’s  financial  and  technical  support  in  the 
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short  and  long  terra.  The  strategic  plan  should  include  a  monitoring  and  evaluation 
framework  for  the  system  as  well  as  an  articulated  workforce  development  plan. 
Financial  support 

Implemented  must  ensure  adequate  financial  support  for  the  system  in  order  to 
build  the  capacity  needed  for  surveillance.  The  workshop  experts  recommended  that 
donors  provide  resources  to  build  the  capacity  needed  and  empower  the  surveillance 
community  in  the  country,  thus  avoiding  the  problem  of  creating  dependency  on  donors 
in  the  long  term. 

Human  resources 

A  successful  surveillance  system  requires  an  appropriately  trained  and  available 
workforce.  System  implementers  needed  to  identify  the  skill  sets  necessary  to  ensure  the 
development  of  a  surveillance  team  that  was  capable  of  producing  quality  data.  The 
workshop  experts  advocated  that  implementers  consider  the  technical,  personality, 
organizational  and  cultural  aspects  of  human  resources  development.  They  related  that 
maintaining  a  permanent  training  program  at  the  regional  level  has  proven  important  in 
ensuring  a  well-trained  workforce  at  every  level  despite  the  reality  of  the  high  turnover  of 
personnel  that  frequently  occurs  in  developing  countries. 

Training  must  be  designed  to  address  a  lack  of  capacity  at  different  levels  and  for 
different  skills.  Workshop  experts  said  that  one  of  the  main  surveillance  training  gaps  in 
the  developing  world  is  training  related  to  data  analysis  in  order  to  enable  surveillance 
staff  to  understand  the  data  collected.  Adequately  trained  data  analysis  managers  become 
critical  as  data  become  more  available. 

Performance  improvement  plan 
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The  presence  of  a  performance  improvement  plan  was  considered  essential  by 
workshop  experts  in  order  to  nurture  a  culture  that  motivated  surveillance  staff.  The 
experts  recommended  using  non-financial  incentives  in  an  effort  to  prevent  the 
sustainability  issues  that  tended  to  occur  when  monetary  incentives  were  used.  One  of  the 
successful  experiences  that  an  expert  shared  was  the  creation  of  a  certain  level  of  healthy 
competition  among  surveillance  staff  by  annually  acknowledging  and  awarding  the  health 
care  facility  that  performed  at  the  highest  level.  Providing  regular  feedback  to  end  users 
of  the  system  has  also  been  shown  to  motivate  surveillance  staff. 

Flexibility  and  adaptability 

All  aspects  of  the  system  needed  to  be  flexible  and  adaptable.  Personnel  and 
software  needed  to  be  flexible  and  adapt  to  changing  environments.  The  system  should 
have  the  ability  to  incorporate  new  or  different  data  sources. 

Intuitive  to  end-users 

Workshop  experts  recommended  that  disease  surveillance  systems  be  intuitive  to 
end-users.  In  their  view,  the  surveillance  staff  at  the  local  level  needed  to  be  able  to  use 
the  system  without  having  an  information  technology  degree.  At  the  same  time,  the  staff 
should  have  sufficient  tools  to  conduct  a  complex  analysis  of  data  if  needed.  The  system 
should  to  be  simple  to  use  and  must  prioritize  data  collection  and  access  processes. 

Useful  at  all  levels 

Surveillance  systems  must  be  useful  at  all  levels,  from  local  users  up  to  the 
national  level.  Implemented  should  avoid  creating  a  parallel  system  if  a  working  disease 
surveillance  system  is  already  in  place.  The  workshop  experts  recommended  working 
within  existing  disease  surveillance  systems  if  such  systems  are  available  and  useful. 
When  developing  a  system,  implemented  should  stay  close  to  system  users  and  listen  to 
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what  they  need.  An  important  means  of  emphasizing  the  usefulness  of  the  system  is  to 
provide  feedback  on  the  information  generated  by  the  system  to  all  stakeholders. 

Data  transparency 

Implemented  must  ensure  data  transparency  by  developing  data  use  agreements 
and  facilitating  data  sharing  between  and  among  jurisdictions.  Implemented  must 
guarantee  that  all  such  sharing  will  be  conducted  legally  and  ethically. 

Start  small 

The  final  recommendation  by  the  panel  of  experts  participating  in  the  workshop 
was  to  encourage  implemented  to  begin  on  a  small  scale.  In  that  way,  implemented  can 
affirm  that  a  given  surveillance  system  works  and  that  they  have  made  all  necessary 
changes  before  implementing  it  on  a  larger  scale.  The  factors  related  to  successful 
implementation  of  electronic  disease  surveillance  systems  in  developing  countries,  agreed 
on  by  workshop  participants  are  summarized  in  Table  1. 

Data  Triangulation 

When  the  researcher  compared  the  factors  agreed  on  during  the  expert  workshop 
with  those  cited  in  the  literature  reviewed  in  the  previous  chapter,  she  found  similarities 
among  all  the  components  of  the  surveillance  systems.  Disease  surveillance  systems  are 
relying  more  on  electronic  tools  in  order  to  develop  more  accurate  and  efficient  ways  to 
collect,  translate,  and  communicate  information.  In  their  surveillance  efforts  that  are 
generally  supported  by  external  organizations,  developing  countries  are  gaining 
increasing  access  to  this  technology.  Technology,  however,  is  not  the  most  important 
component  of  a  surveillance  system.  Country  needs  for  surveillance  must  be  clearly 
articulated  and  government  commitment  should  be  present  from  the  beginning  if  there  is 
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to  be  any  hope  for  success  in  the  surveillance  effort.  Donor  support  needs  to  be  explicitly 
stated  in  terms  of  not  only  the  financial  resources  commitment,  but  also  the  type  and 
extent  of  technical  support  that  the  donor  will  offer. 

Even  when  the  aforementioned  factors  are  considered,  the  process  of 
implementing  a  disease  surveillance  system  is  challenging.  Triangulation  of  the  data 
from  the  review  of  the  scholarly  and  gray  literature  and  the  panel  of  experts  who 
participated  in  the  workshop  emphasizes  the  importance  of  surveillance  system 
implementation  as  a  continuing  learning  process.  Lessons  learned  during  the  process  can 
be  used  to  shape  and  fine-tune  a  continually  evolving  system.  System  fit  and 
sustainability  play  key  roles  in  the  implementation  of  electronic  disease  surveillance 
systems. 
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CHAPTER  5:  RESULTS  -  PRIMARY  DATA  FROM  PERU 


Introduction 

In  this  chapter,  the  researcher  describes  the  proposed  implementation  model 
developed  to  explain  successful  implementation  of  electronic  disease  surveillance 
systems  in  Peru.  In  order  to  address  this  aim,  the  researcher  used  the  theoretical  approach 
of  Grounded  Theory  to  collect,  analyze  and  interpret  primary  data  from  individuals  who 
had  direct  experience  with  surveillance  at  various  organizational  levels.  The  Peruvian 
model  derived  from  these  data  was  influenced  by  both  the  successful  and  unsuccessful 
experiences  of  the  research  participants.  The  researcher  then  engaged  in  a  process  of  data 
triangulation  using  the  Peruvian  electronic  disease  surveillance  model  that  emerged  from 
the  grounded  theory  approach,  the  gray  literature  review,  and  findings  from  the  expert 
consensus  workshop.  In  this  chapter  the  grounded  theory  approach  will  be  described  and 
findings  presented,  including  the  Peruvian  electronic  disease  surveillance  model  that 
emerged  from  the  data. 

Study  Context 

Peru  is  a  South  American  country  with  almost  30  million  inhabitants.  It  is 
bordered  on  the  north  by  Ecuador  and  Colombia,  on  the  south  by  Chile  and  Argentina,  on 
the  east  by  Brazil  and  Bolivia  and  on  the  west  by  the  Pacific  Ocean.  It  is  a  multi-ethnic 
country,  with  Spanish  being  its  main  language.  The  geography  of  the  country  is  quite 
diverse  with  three  recognized  regions:  a  narrow  and  plain  coast,  highlands  with  mountain 
peaks  reaching  20.000  feet,  and  a  wide  jungle  region  that  covers  almost  60%  of  the 
country. 


80 


Ministry  of  Health  (MOH) 

The  Ministry  of  Health  (MOH)  is  the  national  authority  for  all  health  care 
facilities  in  Peru,  including  public  and  private  facilities,  social  security  facilities,  and 
Armed  Forces  health  facilities.  At  the  time  of  this  study,  each  of  the  Armed  Forces 
Health  Departments  functioned  separately  and  would  coordinate  with  the  MOH  as 
needed  or  desired.  The  General  Division  of  Epidemiology  (GDE)  was  the  organization  in 
charge  of  overseeing  disease  surveillance  for  the  MOH.  Nevertheless,  the  National 
Institute  of  Health  (NLH)  within  the  MOH  was  the  main  public  laboratory  in  the  country. 
The  NIH  processed  blood  and  other  human  samples  for  the  diagnosis  of  diseases  that 
were  under  national  surveillance  when  the  regional  laboratories  do  not  have  the  capability 
to  test  them.  The  MOH  also  requested  support  with  laboratory  testing  when  necessary  to 
NAMRU-6.  The  NIH  worked  as  an  organization  separate  from  other  health-related 
organizations  and  includes  a  disease  surveillance  system  based  on  laboratory  diagnosis. 

At  the  time  that  data  were  collected  for  this  study,  the  NIH  surveillance  system  had  not 
been  integrated  with  the  system  developed  by  the  GDE. 

General  Division  of  Epidemiology  Sur\'eillance  System 

The  GDE  had  established  a  network  across  Peru  at  several  organizational  levels  of 
surveillance:  national,  regional,  network,  micro-network  and  local.  The  system  involved 
almost  5,000  reporting  units  in  hospitals,  health  care  centers  and  health  care  posts.  An 
electronic  tool  called  NOTI-SP  was  in  place  for  the  mandatory  notification  of  55  diseases 
or  health  related  events  and  a  national  directive  required  all  GDE  sites  to  comply  with 
notification.  Disease  surveillance  notification  reports  were  typically  sent  weekly  as  either 
individual  data  or  data  that  had  been  grouped  according  to  age  range.  Individual  data 
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reporting  refers  to  the  notification  of  a  disease  or  health-related  event  for  each  diagnosed 
patient  who  presents  to  a  health  care  facility.  The  individually  focused  reports  provided 
detailed  information  for  each  case  such  as  age,  residential  district,  gender,  and  laboratory 
data.  For  instance,  dengue  fever  and  hepatitis  B  were  reported  individually.  Grouped 
reporting  referred  to  diseases  or  other  health-related  data  that  are  summarized  according 
to  age  groups.  This  type  of  reporting  is  usually  assigned  to  diseases  or  health-related 
events  that  occurred  on  a  more  frequent  basis,  such  as  acute  diarrheal  disease  or  sexually 
transmitted  infections. 

The  process  of  reporting  involved  the  reporting  staff  using  NOTI  software  or 
paper  forms  if  a  computer  was  not  available.  The  NOTI  notification  report  flowed  from 
the  local  to  the  micro-network  level  and  then  to  network  surveillance  staff  who 
subsequently  forwarded  the  information  to  personnel  in  the  regional  epidemiology  office. 
Regional  office  staff  consolidated  the  information  and  sent  it  to  the  national  level.  At  the 
national  level,  the  data  were  used  to  create  the  epidemiological  analysis  for  the  country 
and  produce  a  weekly  bulletin  that  was  forwarded  to  all  surveillance  units  in  the  country. 

The  GDE  had  developed  surveillance  subsystems  in  an  effort  to  acquire  more 
information  about  certain  diseases.  The  selection  of  these  diseases  was  based  on  their 
public  health  relevance,  such  as  outbreaks  or  health  emergencies;  nosocomial  infections; 
international  surveillance;  risk  factors;  HIV/AIDS  surveillance;  neonatal  and  perinatal 
surveillance;  Measles,  Rubella,  and  other  febrile  rash  diseases;  and  Syndromic 
Surveillance.  The  syndromes  under  surveillance  for  the  Peruvian  MOH  are  the  following: 
fever,  acute  icteric  fever,  hemorrhagic  fever,  febrile  respiratory  illness,  fever  with 
neurological  symptoms,  eruptive  fever,  acute  diarrhea,  necrotic  skin  ulcer,  acute  anemic 
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fever,  and  unexplained  death  post  fever.  Each  one  of  these  subsystems  had  a  national 
directive  supporting  them  and  involved  the  collection  of  information  through  the  use  of 
different  forms.  Some  diseases  required  the  collection  of  laboratory  samples.  At  the  time 
of  this  study,  the  MOH  surveillance  units  had  several  surveillance  systems  with  which 
they  needed  to  work. 

Sentinel  Surveillance  System 

In  addition  to  the  aforementioned  subsystems,  the  GDE  has  instituted  a  sentinel 
surveillance  system  to  improve  the  timeliness  of  the  information  received.  This  type  of 
surveillance  took  advantage  of  an  online  tool  and  had  been  implemented  for  influenza, 
other  respiratory  viruses,  and  acute  severe  respiratory  infections.  By  using  an  electronic 
tool  for  reporting  data,  the  information  collected  could  be  instantly  accessed  at  the 
regional  and  national  level,  thus  reducing  the  time  that  it  took  for  information  to  reach 
decision  makers.  The  GDE  selected  21  sites  across  the  country  for  the  reporting  of  flu 
syndrome  and  7  hospitals  for  the  surveillance  of  severe  acute  respiratory  infection  (SARI) 
and  mortality  due  to  SARI.  Since  the  intent  of  the  current  study  was  to  investigate  the 
implementation  of  electronic  disease  surveillance  systems  in  Peru,  the  GDE  sentinel 
electronic  surveillance  system  was  one  of  three  systems  included  in  the  study.  The  other 
two  electronic  systems  included  in  the  study  were  the  system  used  by  the  Peruvian  Navy 
and  Peruvian  Army. 

Peruvian  Armed  Forces 

The  Armed  Forces  in  Peru  included  the  Army,  Navy,  and  Air  Force.  The  Navy 
was  the  First  of  the  three  military  services  to  implement  a  disease  surveillance  system.  No 
previously  existing  mandatory  notification  system  had  been  in  place  in  any  of  the 
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services  when  the  Navy  initiated  their  system.  Implementation  of  the  system  was  made 
possible  through  the  support  of  the  US  Naval  Medical  Research  Unit-6  (NAMRU  6)  in 
2003.  A  private  company  contracted  by  the  Navy  developed  the  electronic  surveillance 
tool  used  for  the  Navy’s  system.  The  tool  enabled  reporting  staff  to  send  disease 
notifications  electronically  by  using  an  Internet  connection  or  by  calling  a  toll  free  phone 
number.  The  data  were  collected  for  forty-five  conditions  (diseases  and  syndromes), 
including  mandatory  diagnoses  required  by  the  Peruvian  MOH  and  conditions  of  military 
relevance.  The  surveillance  process  involved  surveillance  personnel  in  initially  recording 
individual  and  grouped  disease  cases  on  pre-coded  forms. 

In  2012,  the  Peruvian  Navy  had  121  active  reporting  units,  which  included  ships 
that  navigated  along  the  Pacific  coast  and  the  rivers  in  the  Amazon  rainforest.  The 
Peruvian  Army  decided  to  implement  a  disease  surveillance  system  using  this  same  tool 
in  2005.  By  2012,  the  Peruvian  Army  had  153  reporting  units  across  the  country.  Both 
the  Army  and  Navy  systems  were  implemented  by  a  surveillance  team  at  the  national 
level,  with  information  flowing  directly  from  the  reporting  unit  to  the  national  level.  This 
national  surveillance  team  was  comprised  of  Peruvian  Navy  and  Army  epidemiologists 
assigned  to  oversee  this  process.  At  the  same  time,  NAMRU-6  created  a  surveillance  hub 
composed  of  public  health  professionals  to  provide  technical  and  financial  support.  They 
also  offered  office  space  and  logistics  for  one  representative  of  the  Public  Health 
Department  associated  with  each  of  the  Armed  Forces.  Their  presence  was  intended  to 
provide  an  environment  needed  to  monitor  the  system  and  to  facilitate  the  coordination  of 
the  surveillance  process  within  the  Armed  Forces. 
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The  characteristics  of  the  three  surveillance  systems  under  study  have  been 
summarized  in  Table  2. 


85 


Table  2:  Characteristics  of  the  surveillance  systems  studied  to  develop  the  Peruvian 
Electronic  Disease  Surveillance  Systems  Implementation  model. 


System 

Influenza 

Alerta  DISAMAR 

Vigila  COSALE 

Institution 

Ministry  of  Health 
(MOH) 

Peruvian  Navy 

Peruvian  Army 

Type  of 
surveillance 

Sentinel 

National 

National 

Year  started 

2005 

2003 

2005 

#  of  surveillance 
units 

52  sites 

121  sites 

153  sites 

Location  of  the 
sites 

Capital  and  main 
cities  (urban) 

Urban  and  rural 
sites,  and  shipboard 

Urban  and  rural 
sites 

Events  under 
surveillance 

Outpatient  diagnosis 
of  flu-like  syndrome 
AND  hospitalized 
patients  for 
respiratory 
syndrome  and  severe 
acute  respiratory 
infection 

45  diseases  and 
syndromes 

45  diseases  and 
syndromes 

People  covered  by 
the  system 

Data  not  available 
from  MOH. 

Usage  rate  for 
hospitals  in  urban 
areas  in  Peru  is  65%; 
usage  rate  for  lower 
level  health  facilities 
in  urban  areas  in 

Peru  is  57%  (62). 

128,000  people 
including  navy 
personnel  and 
family  members 

384,000  people 
including  army 
personnel  and 
family  members 

Products  of  the 
system 

Daily  and  weekly 
notification  of 
events 

Weekly  notification 
of  diseases  and 
syndromes 

Weekly  notification 
of  diseases  and 
syndromes 

Financial  support* 

Primarily  funded  by 
MOH  with  support 
of  external 
organizations  for 
training  sessions  and 
equipment 

Technology**, 
technical  assistance, 
and  training  funded 
and  still  provided  by 
the  US  Naval 

Medical  Research 
Unit  6  in  Lima  Peru 

Technology**, 
technical  assistance, 
and  training  funded 
and  still  provided  by 
the  US  Naval 

Medical  Research 
Unit  6  in  Lima  Peru 

♦Costing  data  was  unavailable  for  any  of  the  systems. 
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**  Technology  costs  have  dramatically  decreased  last  year  due  to  the  migration  to  an 
open  source  platform 

Definitions  of  Terms 

Throughout  this  chapter,  the  researcher  will  use  the  term  “implemented"  to  refer 
to  the  national  surveillance  team  that  led  the  implementation  process  and  the  technical 
staff  from  external  organizations  that  supported  the  process. 

The  researcher  used  the  term  “institution”  to  refer  to  the  Peruvian  Ministry  of 
Health  or  the  Armed  Forces;  while  the  term  “organization"  was  used  to  refer  to  the 
organizations  within  the  larger  institutions.  For  the  Armed  Forces  institution,  the 
researcher  included  the  Peruvian  Navy  and  the  Peruvian  Army  organizations  in  this 
study.  For  the  MOH  institution,  the  researcher  described  the  General  Division  of 
Epidemiology  and  the  National  Institutes  of  Health  organizations. 

The  term  “surveillance  tool"  was  used  to  describe  the  software  or  platform 
developed  for  the  purposes  of  data  collection,  transmission  and  analysis  in  a  disease 
surveillance  system. 

The  researcher  used  the  term  “surveillance  unit  or  site"  or  “reporting  unit  or  site” 
to  refer  to  any  hospital,  health  care  center,  health  care  post,  or  military  unit  that  employed 
one  or  several  health  care  workers  trained  to  fulfill  surveillance  tasks  in  addition  to  their 
regular  tasks.  Due  to  the  extensive  scope  and  depth  of  the  MOH  surveillance  program, 
the  MOH  units  had  greater  personnel  capacity  to  perform  surveillance  tasks  compared  to 
the  Armed  Forces.  However,  existing  surveillance  staff  limitations  prohibited  the 
dedication  of  one  person  exclusively  to  surveillance  reporting  tasks.  For  the  Armed 
Forces,  the  number  of  personnel  in  a  given  surveillance  site  ranged  from  hundreds  in  a 
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central  military  hospital  to  one  combat  unit-based  health  care  worker  who  was  in  charge 
of  all  health  and  surveillance-related  activities  in  the  unit. 

Disease  surveillance  implementation  model 

The  proposed  implementation  model  was  developed  after  extensive  in-depth 
interviews  with  34  stakeholders  involved  with  the  successful  implementation  of  the 
influenza  sentinel  disease  surveillance  system  at  the  MOH  and  the  mandatory  national 
disease  surveillance  system  at  the  Peruvian  Navy  and  the  Peruvian  Army.  The 
interviewees  were  members  of  different  stakeholders  groups:  reporting  staff,  national 
epidemiology  team,  regional  epidemiology  team,  high-level  administrators,  and  external 
technical  assistance  providers  (Appendix  L).  In-depth,  open-ended  qualitative  interviews 
took  place  in  Lima  and  Iquitos,  Peru,  from  February  to  April  2012.  Follow-up 
conversations  were  conducted  over  the  phone.  In  total,  23  interviews  were  conducted, 
ranging  from  30  minutes  to  one  hour  and  50  minutes  in  duration.  The  researcher  conducted 
all  interviews  in  Spanish  since  it  was  the  interviewees’  primary  language  and  the  researcher 
was  a  native  Spanish  speaker. 

Research  findings  revealed  that  the  successful  implementation  of  an  electronic 
disease  surveillance  system  required  the  presence  of  several  factors  at  each  of  the 
following  levels:  a)  the  administrative  level  within  an  organization,  b)  the  national  and 
regional  epidemiology  team  level,  and  c)  the  surveillance  unit  or  site  level.  The  analysis 
of  primary  data  using  grounded  theory  methodology  led  to  themes  and  concepts  that  the 
researcher  presented  visually  in  Figure  6. 
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Institutional 

Administrative 


r  ” 


National  and 
Regional 
Surveillance 
Team 


•Organizational  need 

•Organizational  commitment 

•Legislative  action  (content,  flexibility,  valnei 

•Financial  support  (perceived  ownership,  staff  expectiotis) 

•Technical  assistance 

•Coordination 

•Value  and  action  (disease  detection  and  rapid  response;  planning, 
organizing,  and  prevention) 


•Surveillance  staff  characteristics  (staff  turnover,  workload,  limited 
surveillance  staff  skills) 

•Surveillance  feasibility  (simplicity,  standardization,  flexibility) 
•Initiating  on-site  surveillance  team  (training) 

•Surveillance  staff  motivation  (surveillance  awareness,  surveillance 
site  resources) 


•Team  characteristics  (knowledge  and  skill,  availability,  continuing 
education) 

•Team  role  (monitoring,  feedback,  supervision,  support) 


Figure  6:  Three  implementation  levels  identified  through  analysis  of  primary  data  using 
Grounded  theory  methodology. 

Point  of  Initiation:  Institutional  Administrative  Level 

Findings  revealed  that  the  successful  implementation  of  an  electronic  disease 
surveillance  system  within  an  organization  initially  depended  on  its  high-level  authorities 
recognizing  the  need  for  and  then  committing  themselves,  their  organization,  and 
organizational  resources  to  system  implementation.  Their  recognition  of  a  clear  need  for 
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a  system  tended  to  facilitate  their  political  will  to  take  action  to  initiate  the  system.  It  was 
necessary  for  them  to  translate  their  commitment  into  two  related  actions  that  culminated 
in  system  initiation  and  sustainability:  a)  the  generation  of  legislative  action  that 
supported  all  activities  and  processes  associated  with  the  system,  and  b)  the  dedication  of 
financial  and  other  resources  necessary  to  not  only  initiate,  but  also  sustain  the  system. 

Due  to  the  limited  resources  dedicated  to  disease  surveillance  as  well  as 
bureaucratic  restrictions  on  the  use  of  these  funds  to  buy  certain  equipment  or  pay  for 
services  at  government  institutions,  administrators  sometimes  sought  additional  financial 
support  for  system  implementation  through  external  donors  (Int  8,  9).  The  technical 
support  required  for  the  implementation  process  came  from  within  the  initiating 
organization  or  through  other  international  or  foreign  entities  (Int  8,  9).  At  the 
administrative  level  of  the  organization,  it  was  crucial  for  high-level  authorities  to 
establish  mechanisms  of  coordination  between  organizations  and  within  the  different 
levels  of  the  system,  and  to  develop  collaborative  networks  across  institutions  (Int  8,  9). 

The  outline  of  the  concepts  identified  at  this  level  is  shown  in  Table  3. 
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Table  3:  Outline  of  the  high  administrative  level  concepts  in  the  Peruvian  electronic 
disease  surveillance  system  implementation  model,  as  identified  by  the  primary  data 
analysis. 


Institutional/Organizational  need 
Institutional/Organizational 
commitment 
Legislative  Action 


Financial  support 

Technical  assistance 
Coordination 


•  Directive  content 

•  Directive  flexibility 

•  Directive  value 

•  Perceived  ownership 

•  Staff  expectations 

•  .Among  different  organizations  within  the 
same  institution 

•  Between  external  technical  support  and 
the  organization 

•  Between  organizations  affiliated  with 

different  organizations _ 


Institutional/Organizational  Need 

Rresearch  participants  reported  that  any  successful  disease  surveillance  system 
must  be  developed  and  implemented  in  response  to  a  clear  need  by  the  target  institution 
or  organization.  The  electronic  disease  surveillance  systems  that  the  Peruvian  Navy  and 
Army  deemed  successful  were  their  first  national  surveillance  systems.  The  Navy  had 
clearly  identified  a  need  for  a  surveillance  system  after  a  malaria  outbreak  occurred  in  a 
remote  area  of  the  rainforest.  The  lack  of  a  surveillance  system  prevented  naval  public 
health  authorities  from  detecting  and  responding  to  the  malaria  outbreak  quickly  and 
effectively  (Int  2,  Int  7,  Int  13).  The  Peruvian  Army  had  similar  experiences,  mainly  in 
remote  areas  where  their  military  missions  took  place  (Int  1 1).  For  the  GDE,  the 
influenza  sentinel  surveillance  system  was  implemented  in  response  to  a  pandemic 
influenza  threat  (Int  8,  Int  9). 
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Research  participants  observed,  in  contrast,  that  lacking  a  clear  need  for  a 
surveillance  system  could  contribute  to  a  failure  of  a  surveillance  system's 
implementation  process.  For  instance,  NAMRU-6  offered  the  same  data  collection 
instrument  for  surveillance  that  had  been  successfully  implemented  in  the  Peruvian 
Armed  Forces  to  the  Ecuadorian  Armed  Forces  and  the  MOH  in  Panama.  The 
surveillance  tool  was  implemented  in  Panama  and  Ecuador  but  later  abandoned  by  both 
institutions  (Int  1,  Int  7);  in  part  because  both  had  an  existing  disease  surveillance  system 
and  did  not  entirely  depend  on  the  information  collected  by  this  tool  to  make  their  public 
health  decisions.  For  the  MOH  in  Panama,  public  health  authorities  in  that  country 
viewed  the  NAMRU-6  tool  as  complementary  and  designed  only  to  report  outbreak 
situations  (Int  7). 

Institutional/Organizational  Commitment 

While  organizational  need  was  viewed  as  a  starting  point,  research  participants 
viewed  an  organizational  or  institutional  commitment  to  system  implementation  as 
critical.  Once  an  organization  or  institution  demonstrated  a  clear  need  for  an  electronic 
disease  surveillance  system,  research  participants  in  the  study  indicated  that  the  next  step 
in  implementing  a  system  was  to  obtain  commitment  from  high  level  authorities  at  the 
organization  or  the  institution  where  the  system  would  be  implemented.  These 
individuals  needed  to  be  not  only  committed  to  electronic  disease  surveillance  system 
implementation,  but  also  to  the  goal  of  integrating  the  system  into  the  organization  or 
institution  and  to  developing  and  implementing  a  strategy  to  achieve  that  purpose. 

Two  research  participants  viewed  institutional  commitment  as  a  particularly 
important  issue  when  the  surveillance  system  was  to  be  implemented  in  collaboration 
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with  an  external  funding  agency  or  through  external  technical  assistance.  External 
funding  and  technical  assistance  tended  to  be  time-limited  and  could  contribute  to 
surveillance  system  failure  due  to  a  lack  of  longer-term  sustainability  since  integration  of 
the  system  into  the  organization  was  unlikely  to  occur  (Int  1).  As  one  research  participant 
explained, 

De  nada  sirve  implementor  un  sistema  super  de  vigilancia ,  si  pues  el  director  ni 
el  ministro  van  a  apoyar  esto.  Sin  tampoco  el  marco  normal ivo  correspondiente, 
porque  tiene  que  estar  esto  en  un  marco  normativo;  sin  los  mecanismos  de 
fmanciacion  tambien  correspondiente.  (Int  20) 

[English  translation]  It  is  not  of  any  use  to  implement  a  super  disease  surveillance 
system ,  if  the  director  or  the  minister  would  not  support  it.  Without  legislation 
that  supports  the  system,  because  the  system  has  to  have  supporting  legislation. 
The  system  also  needs  the  financial  mechanisms  to  function. 

In  the  presence  of  organizational  or  institutional  leadership  support  and  a  clear 
awareness  of  the  importance  of  and  need  for  such  a  system,  research  participants  believed 
that  commitment  to  the  system  could  readily  be  achieved  (Int  8,  Int  10,  Int  3,  Int  20,  Int 
21).  For  the  Armed  Forces,  however,  commitment  to  implementation  presented 
additional  challenges  because  their  leadership  often  lacked  a  public  health  background. 
National  epidemiology  officers  then  faced  the  challenge  of  raising  the  awareness  of  high- 
level  Armed  Forces  authorities  about  disease  surveillance  and  convincing  them  about  the 
benefits  and  need  for  such  systems  (Int  22). 

Organizational  or  institutional  commitment  to  the  system  was  not  only  necessary 
to  sustain  support  for  the  system,  but  also  to  obtain  adequate  resources,  enact  legislation 
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in  support  of  the  system,  and  facilitate  key  implementation  actions  or  activities  (Int  1 1 , 

Int  13,  Int  7,  Int  3,  Int  15).  Implementation  actions  included  seeking  approval  for 
personnel  to  attend  training  courses  (Int  15);  developing  logistics  for  training  sessions 
and  supervision  visits  (Int  18);  facilitating  the  logistics  for  the  surveillance  processes  such 
as  the  use  of  computers  for  reporting  (Int  3);  transporting  samples  to  the  central 
laboratory  (Int  22);  and  supporting  policies  and  procedures  that  needed  to  be  in  place  in 
order  for  personnel  to  respond  quickly  and  effectively  to  the  disease  events  detected  by 
the  surveillance  system  (Int  17). 

Key  leaders  sometimes  played  a  critical  role  in  surveillance  activities  such  as 
training  sessions  (Int  2)  or  raising  awareness  during  ceremonial  presentations.  For 
instance,  during  a  national  training,  the  Army  Surgeon  General  gave  a  speech  to  all 
personnel  about  the  importance  of  electronic  disease  surveillance  systems.  Research 
participants  in  this  study  believed  that  the  speech  contributed  to  the  high  morale  of 
surveillance  staff  and  to  an  increased  awareness  by  line  officers  and  enlisted  personnel 
about  the  importance  of  disease  surveillance  in  general  (Int  17,  Int  21 ). 

If  the  head  of  an  organization,  region,  district,  hospital  or  surveillance  site  was 
aware  of  the  importance  of  the  system  and  supported  its  implementation,  it  was  more 
likely  that  staff  under  his/her  direction  would  follow  suit  (Int  18,  Int  22,  Int  23,  Int  20). 
Research  participants  mentioned  examples  at  every  level,  starting  from  the  Surgeon 
General  or  Minister  of  Health  (Int  20,  Int  3,  Int  21),  and  going  through  the  Director  of  the 
National  Epidemiology  Office  (NEO)  (Int  20,  Int  5,  Int  22),  Director  of  the  NEO 
Surveillance  Office  (Int  7),  Director  of  the  Regional  Office  of  Epidemiology  (Int7,  Int 
23),  and  Director  of  the  Surveillance  Site  (Int  22,  Int  3)  (Figure  7). 
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Figure  7:  Hierarchy  of  disease  surveillance  authorities. 


Legislative  Action 

One  of  the  most  important  elements  needed  once  the  organization  or  institution’s 
commitment  was  achieved,  was  the  enactment  of  legislative  action.  Several  research 
participants  viewed  national  legislative  action  to  develop  a  national  directive  as  a  key  to 
the  success  of  a  surveillance  system  (Int  1,  Int  10,  Int  20,  Int  23,  Int  4,  Int  5,  Int  8). 
Developing  a  national  directive  based  on  the  legislation  was  among  the  initial  steps  that 
research  participants  in  the  study  mentioned  in  the  context  of  implementing  an  electronic 
disease  surveillance  system  (Int  1 3).  Legislation  and  a  national  directive  meant  that 
health  workers  and  other  staff  assumed  responsibility  for  performing  the  identified 
surveillance  tasks.  It  also  signified  that  surveillance  staff  could  be  held  accountable  for 
their  actions  or  inactivity  once  they  had  been  provided  with  the  tools  and  training  to  use 
the  electronic  surveillance  system  (Int  13,  Int  2,  Int  8). 

For  military  organizations,  a  directive  also  represented  a  tool  that  implementers 
could  use  to  position  the  system  as  an  integral  part  of  the  organization.  The  dissemination 
of  these  documents  to  all  the  surveillances  sites  was  important  to  ensure  that  the 
commanders  at  the  sites  provided  the  support  to  the  surveillance  staff  for  the  fulfillment 
of  the  surveillance  tasks  described  in  this  document  (Int  2,  Int  4).  One  research 
participant  associated  system  failure  with  a  lack  of  national  legislation  or  a  directive 
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regarding  a  disease  surveillance  system  in  the  Armed  Forces  in  Ecuador  (Int  I ). 
According  to  three  research  participants,  it  would  be  difficult  to  find  the  support  needed 
to  execute  the  activities  required  for  surveillance  system  implementation  in  the  absence 
of  legislation  or  a  directive  (Int  1,  Int  10,  Int  9). 

Content 

As  important  as  it  was  to  have  legislation  and  a  national  directive,  research 
participants  believed  it  was  equally  important  to  include  clear  and  precise  guidelines 
within  the  directive  (Int  23,  Int  7).  In  their  view,  the  directive  should  specify  the  system’s 
components,  all  the  steps  of  the  disease  surveillance  processes,  and  the  resources  required 
to  accomplish  all  surveillance  tasks  (Int  10,  Int  20,  Int  2,  Int  8,  Int  9).  They  underscored 
the  necessity  for  implementers  to  follow  the  directive.  They  should,  for  example,  design 
surveillance  activities  according  to  what  was  described  in  the  directive  (Int  9).  If  it  was 
not  written  in  the  directive,  support  for  necessary  surveillance  activities  was  difficult  to 
marshal  (Int  6).  On  the  other  hand,  when  a  task  was  included  in  the  directive,  surveillance 
staff  would  be  able  to  request  the  resources  required  for  implementation.  Such  resources 
might  he,  for  example,  computers  for  use  in  an  information  technology  (IT)  department 
(Int  4).  In  addition,  national  or  regional  surveillance  team  leaders  could  reference  the 
legislation  or  national  directive  when  sending  congratulatory  messages  to  personnel  who 
had  exhibited  exemplary  work  performance.  Such  congratulatory  messages  served  as  a 
means  of  acknowledging  excellent  work  and  as  a  potential  motivator  for  continued 
employee  excellence  (Int  21 ). 

Flexibility 

Research  participants  observed  that  flexibility  and  responsiveness  to  change  in 
disease  surveillance  systems  and  in  the  directive  or  legislation  that  guided  them  would 


96 


facilitate  the  implementation  process.  A  change  in  the  public  health  situation  in  Peru,  for 
instance,  may  necessitate  a  change  in  the  country's  disease  surveillance  system.  Once  a 
directive  was  published  or  legislation  enacted,  therefore,  system  implementers  had  to  be 
able  to  adapt  it  to  changing  circumstances  in  a  timely  manner  (Int  1,  Int  9). 

Value 

The  existence  of  legislation  related  to  disease  surveillance  virtually  guaranteed 
continuity  in  the  organization’s  commitment.  Regardless  of  changes  in  high  level 
administrative  positions,  for  example,  existing  legislation  or  a  directive  meant  that 
established  disease  surveillance  priorities  would  continue  to  be  maintained  and  supported 
(Int  1,  Int  3,  Int  5.  Int  7). 

Financial  Support 

The  second  key  element  that  could  be  obtained  in  support  of  a  system,  given 
institutional  or  organizational  commitment,  was  financial  support.  Research  participants 
identified  the  designation  of  adequate  financial  resources  in  the  implementation  of  a 
surveillance  system  as  a  critical  need.  Ideally  such  resources  would  come  from  the 
organization  where  the  system  was  to  be  implemented  (Int  10,  Int  20).  However,  due  to 
the  limited  resources  for  disease  surveillance  in  countries  like  Peru,  it  was  common  for 
disease  surveillance  systems  to  be  implemented  with  the  financial  support  of  an  external 
donor  (Int  13,  Int  1 1,  Int  18,  Int  2.  Int  5,  Int  1,  Int  6), 

Regarding  the  commitment  of  the  organizations  to  invest  in  disease  surveillance, 
the  Armed  Forces  faced  a  different  situation  than  that  which  was  faced  by  the  MOH. 
Military  organizations  claimed  national  security  as  their  number  one  goal,  not  health. 
They  therefore  designated  a  very  small  part  of  their  budget  to  health  with  an  even  smaller 
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piece  of  it  going  to  preventive  medicine  (Int  1 1  ).  The  military  organizations  used  external 
funding  to  develop  their  first  and  only  national  surveillance  system  (Int  1 1.  Int  13). 
Without  this  economical  support,  they  could  have  not  afforded  an  electronic  tool,  and  the 
many  resources  that  the  implementation  process  required  such  as  the  cost  of  training  and 
supervision  (Int  11).  Armed  Forces  authorities  claimed  they  did  not  have  money  available 
to  buy  equipment  such  as  computers  and  pay  for  Internet  services.  They  relied  on 
external  funding  to  buy  the  equipment  they  needed  for  surveillance  purposes  (Int  6). 

For  the  MOH,  the  challenges  were  different.  They  had  a  budget  assigned  for  these 
activities.  However,  their  budget  was  limited,  especially  for  buying  the  required 
equipment.  Of  necessity,  external  organizations,  such  as  the  Pan  American  Health 
Organization  (PAHO),  continued  to  provide  funding  for  equipment,  training,  logistics, 
laboratory  supplies,  infrastructure,  and  the  hiring  of  personnel  (Int  9,  Int  23,  Int  13). 
Moreover,  the  process  to  buy  equipment,  even  small  equipment  such  as  printers,  was 
extremely  lengthy  and  tedious  (Int  23).  Surveillance  staff  had  many  administrative  steps 
to  follow  in  making  such  purchases  and  the  process  sometimes  took  more  than  one  year 
(Int  9).  Therefore,  surveillance  supervisors  who  developed  the  surveillance  tools  for  the 
system  often  faced  limited  technology  when  striving  to  have  the  system  ready  when  it 
was  needed  (Int  8). 

Research  participants’  major  concern  about  the  external  funding  of  surveillance 
systems  was  that  such  funding  was  typically  costly  and  of  limited  duration.  For  instance, 
the  expense  entailed  by  hiring  a  private  company  to  develop,  test,  implement,  and 
periodically  revise  a  surveillance  instrument  for  the  Armed  Forces  was  assumed  by 
NAMRU-6  for  a  decade.  After  10  years,  however,  the  continuation  of  the  external 
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funding  became  unsustainable,  the  Armed  Forces  were  not  able  to  afford  the  cost,  and  a 
solution  was  needed.  In  this  case,  data  from  the  original  tool  were  able  to  be  collected  by 
means  of  an  open  source  tool.  In  the  end,  the  disease  surveillance  system  was  sustained 
only  by  exercising  the  option  of  accessing  a  free  tool  (Int  12,  Int  1,  Int  11). 

Perceived  Ownership 

When  external  organizations  supported  the  implementation  process  of  a  disease 
surveillance  system,  successful  implementation  depended,  in  part,  on  system  staff 
perceiving  ownership  of  the  system  (Int  14).  If  members  of  the  surveillance  team 
believed  that  what  was  asked  of  them  had  originated  from  an  external  organization,  one 
research  participant  indicated  that  such  individuals  might  feel  less  compelled  to  complete 
the  assigned  tasks  (Int  3). 

Staff  expectations 

In  the  case  of  foreign  organizations’  involvement  in  supporting  the  process  of 
disease  surveillance  implementation,  the  staff  recipients  of  their  support  had  expectations 
that  “gifts”  would  accompany  the  surveillance  “project.”  These  expectations  occurred 
due  to  a  long  history  of  how  foreign  organizations  worked  in  Peru.  Foreign  organizations 
had  been  primarily  interested  in  studying  a  specific  disease  and  their  projects  had  funding 
to  buy  equipment  required  for  the  project.  This  equipment  typically  stayed  in  the  country 
after  such  project  ended.  Such  “gifts”  might  have  included  computers  and  printers  for 
staff  use  at  the  surveillance  site  (Int  2).  Therefore,  it  was  important  that  the  organization 
delivered  to  staff  a  clear  message  of  system  ownership,  making  sure  the  surveillance  staff 
understood  that  rather  than  being  externally  owned,  the  system  that  received  external 
support  belonged  to  the  organization. 
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Technical  assistance 


In  addition  to  financial  support  of  the  system,  another  consideration  at  this  level 
was  the  need  for  technical  assistance.  Organizations  such  as  the  MOH  had  extensive 
experience  in  disease  surveillance,  but  were  open  to  receive  technical  support  from  other 
organizations  such  as  PAHO  and  CDC.  In  the  case  of  electronic  disease  surveillance 
program  technical  support,  it  was  the  Armed  Forces  that  experienced  the  greater  need,  to 
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which  the  Peruvian  MOH  and  external  organizations  such  as  a  NAMRU-6  responded. 

If  an  external  organization  provided  technical  assistance  for  the  surveillance 
process,  a  research  participant  raised  the  point  that  the  aim  of  that  organization  should  be 
to  build  capacity  within  the  recipient  organization  in  order  to  develop  a  team  that 
ultimately  would  be  able  to  lead  the  implementation  process  (Int  7).  Technical  support 
should  continue  to  the  point  that  the  organization  was  able  to  administer  the  process 
independently  (Int  7). One  research  participant,  for  instance,  considered  the 
implementation  process  a  failure,  "el  sistema  fracaso,  ”  when  the  organization  took  over 
the  surveillance  process  prematurely  (Int  7). 

Coordination 

Those  who  participated  in  the  study  indicated  that  system  coordination  had  to  be 
clearly  defined  and  established  at  the  highest  administrative  levels  within  institutions  and 
among  and  within  organizations  that  were  a  part  of  institutions.  Institutions  such  as  the 
MOH  had  several  organizations  that  worked  with  disease  surveillance,  including  the 
DGE  and  the  NIH  that  recognized  the  need  to  improve  their  collaboration.  Furthermore,  a 
need  existed  for  the  coordination  of  efforts  within  individual  organizations.  Disease 
surveillance  functions  within  DGE,  for  example,  were  structured  into  several  groups  of 
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diseases  under  surveillance  such  as  zoonotic  diseases,  vaccine  preventable  diseases. 
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diseases  transmitted  by  vectors,  and  others.  Research  participants  believed  that  these 
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groups  would  have  benefitted  from  coordinating  their  surveillance  activities. 

Once  a  system  was  developed,  strategies  had  to  be  developed  to  enact  the 
coordination  of  work  at  the  different  levels  of  the  disease  surveillance  network. 

Therefore,  the  task  at  the  highest  administrative  level  was  to  establish  and  disseminate  the 
coordination  mechanisms  across  the  organizations  within  the  same  institution  by 
involving  groups  at  different  levels  within  the  same  organization.  It  was  also  critical  that 
institutions  developed  partnerships  with  other  institutions  in  order  to  enhance  their 
surveillance  capabilities. 

Among  different  organizations  within  the  same  institution 

Several  organizations  affiliated  with  the  Peruvian  MOH  used  different 
surveillance  systems  to  provide  epidemiological  surveillance  data,  including  laboratory 
and  public  health  data.  Although  each  surveillance  system  collected  information  from  the 
same  patients,  at  the  time  of  this  study  the  systems  did  not  share  the  same  data  (Int  10,  Int 
5).  Research  participants  identified  one  of  the  key  elements  in  the  sustainability  of 
disease  surveillance  as  the  coordination  of  and  eventual  integration  of  these  disparate 
systems  in  order  to  meet  the  need  for  timely  and  more  complete  surveillance  data  (Int  9). 
They  believed  that  effective  communication  channels  had  to  be  established  among 
organizations  within  the  MOH.  A  research  participant  offered  the  following  comment: 

En  el  Ministerio  de  Salud  tenemos  dos,  uno  de  nosotros,  de  lo  que  es  la 
notificacion  en  linea ,  [....]  y  estamos  ahorita  trabajando  una  plataforma,  pero 
necesitamos  un  campo  en  eomun,  que  pueda  permitir  jalar  la  informaeidn  de 


101 


labor atorio  v  vice  versa,  la  informacion  epidemiologica ,  los  laboratories  y  no 
estar  digitando  la  informacion  de  las  fichas  en  dos  lugares  diferentes.  ( bit  9) 
[English  translation]  In  the  Ministry  of  Health  nr  have  two  [surveillance  tools], 
one  that  is  the  online  system  [....]  We  are  now  working  on  a  platform  but  we  need 
a  field  in  common,  that  allows  us  to  pull  the  laboratory  information  and  vice 
versa,  the  epidemiological  information,  the  laboratory  information,  and  not  have 
to  enter  information  on  the  forms  in  two  different  places.  ( hit  9) 

BetM’een  external  technical  support  and  the  organization 

Research  participants  observed  that  the  facilitation  of  on-going  communication 
between  technical  support  staff  situated  outside  the  organization  and  surveillance 
personnel  located  within  the  organization  was  key  to  maintaining  an  effective  problem¬ 
solving  environment.  They  contended  that  even  when  communication  channels  were 
established,  true  collaboration  only  occurred  when  personnel  from  disparate  areas  worked 
closely  together  at  the  onset  of  a  particular  surveillance  process.  In  any  collaborative 
effort  it  was  important  to  appoint  a  process  technical  leader  in  conjunction  with  assigning 
roles  within  the  work  team  (Int  12). 

It  was  quite  important  to  develop  a  relationship  of  trust  between  members  of  the 
external  funding  organization  and  organizational  personnel  that  facilitated  positive 
working  relationships.  In  Peruvian  culture,  it  is  culturally  important  to  become  friends 
before  becoming  business  partners  (Int  7).  Research  participants  described  occasions 
when  tasks  or  goals  were  not  accomplished  because  of  a  lack  of  coordination  of  efforts  or 
political  disagreements  (Int  10,  Int  11,  Int  3,  Int  12,  Int  8).  They  related  successful  system 
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implementation  to  people  being  able  to  listen  to  an  experienced  counterpart  and  to 
making  joint  decisions  (Int  7,  Int  12). 

Between  organizations  affiliated  with  different  institutions 

For  surveillance  staff  in  military  organizations  it  was  critical  to  establish  a 
collaborative  relationship  with  the  MOH,  and  especially  imperative  for  sites  located 
outside  the  capital  of  the  country  where  mutual  support  could  be  offered  if  disease 
outbreaks  occurred.  Research  participants  foresaw  that  the  MOH  and  Armed  Forces 
surveillance  systems  would  ideally  be  integrated  at  some  point  in  the  future  (Int  14,  Int 
15,  Int  11,  Int  4). 

The  concepts  and  relationships  at  the  high  administrative  level  are  summarized  in 


Figure  8. 


Figure  8:  Institutional  administrative  level. 

Surveillance  site-based  human  resources 
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The  second  level  identified  in  this  model  was  the  surveillance  site.  As  described 


earlier,  a  surveillance  site  was  a  health  care  facility  or  a  military  unit  that  had  among  their 
staff  a  health  care  worker  trained  to  fulfill  surveillance  tasks.  At  this  level,  the  human 
resources  were  identified  as  the  key  element  for  the  success  of  the  surveillance  process. 

In  this  section,  the  researcher  will  describe  the  concepts  identified  that  enabled  the  high 
performance  of  the  surveillance  staff  as  well  as  those  that  created  the  greatest  difficulties 
in  their  work. 

While  the  support  of  high  level  authorities  was  viewed  by  research  participants  as 
critical  to  start  the  surveillance  implementation  process,  without  human  resources  at  the 
surveillance  site  that  were  skilled  and  motivated,  a  disease  surveillance  system  could  not 
be  initiated  and  would  not  be  able  to  function.  Based  on  findings  that  emerged  from  this 
study,  the  model  for  successfully  implementing  an  electronic  disease  surveillance  system 
in  Peru  included  human  resources  as  a  fundamental  component.  In  this  section,  the 
researcher  will  explain  human  resources  needs  at  the  surveillance  site  level  and  will 
describe  the  elements  required  at  the  surveillance  site  and  the  national  and  regional  levels 
to  ensure  the  availability  of  adequate  human  resources.  The  sentiments  of  several 
research  participants  about  the  importance  of  surveillance  personnel  were  captured  by  the 
following  research  participants’  comment: 

y  pensar  siempre  que  el  sistema  opera  gracias  a  las  personas  que  estdn  en  las 
unidades  notificantes,  ellos  son  los  elementos  clave  para  el  exito  del  sistema.  (Int 
21) 
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[English  translation:]  Always  think  that  the  system  works  because  of  the  people  in 
the  surveillance  sites;  they  are  the  key  elements  fur  the  success  of  the  system.  ( hit 
21). 

The  outline  of  the  surveillance  site  level  concepts  is  depicted  in  Table  4. 
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Table  4:  Outline  of  the  surveillance  site  level  concepts  in  the  Peruvian  electronic  disease 
surveillance  system  implementation  model,  as  identified  by  the  primary  data  analysis. 


Surveillance  staff  • 

characteristics  • 

Surveillance  feasibility  • 


Initiating  on-site  surveillance  • 

teams 


Surveillance  staff  motivation  • 


Staff  turnover 
Workload 

Limited  knowledge  and  skills 
Simplicity 
Standardization 
Flexibility: 

■  In  responsiveness  to  changing  needs 

■  In  reporting  deadlines 

■  In  reporting  expectations 

■  In  means  of  communicating  reports 

■  In  the  surveillance  tool 

Training 

■  Target 

■  Context 

■  Delivery  strategies 

■  Type  (continuous  and  comprehensive) 

■  Train  the  trainers  approach 

•  Institutionalization  of  training 
Surveillance  awareness 
Surveillance  site  resources 

•  Availability 

•  Administrative  support 

■  Performance-based  access 

■  Existing  resources  use 

■  Incentives 

■  Training  as  motivator 

■  Accountability _ 


Surveillance  Staff  Characteristics 

Surveillance  staff  in  the  systems  studied  always  included,  at  minimum,  a  health 
care  worker  at  the  health  care  facility  or  military  unit.  Surveillance  tasks  are  primarily 
conducted  by  nurses  or  military  nurses  (nurses  who  had  received  their  education 
throughout  the  Armed  Forces  rather  than  a  traditional  college).  However,  young 
physicians  as  well  as  nurse  technicians  were  trained  to  conduct  surveillance  tasks  as  well. 
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As  mentioned  earlier,  depending  on  the  size  of  the  facility,  surveillance  staff  could  vary 
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from  only  one  person  in  most  of  the  combat  units  to  several  workers  at  larger  facilities. 

1 1  i. 

Research  participants  associated  the  development  of  a  skilled,  motivated 
surveillance  staff  with  three  main  challenges:  staff  turnover,  workload  issues,  and  skill 
level.  Each  of  these  challenges  will  be  discussed  in  the  following  section. 

Staff  turnover 

Several  participants  in  this  study  observed  high  surveillance  staff  turnover  as  a 
challenge  that  affected  both  the  Armed  Forces  and  the  MOH  surveillance  programs  hut 
was  noticed  to  a  greater  degree  in  the  Armed  Forces  (Int  1 1,  Int  16.  Int  19.  Int  21,  lnt  23, 
Int  9).  Personnel  in  the  Armed  Forces  were  rotated  once  or  twice  a  year  across  the 
country.  New  personnel  coming  to  the  site  may  not  have  been  trained  or  have  less 
motivation  to  perform  surveillance  tasks.  A  change  in  personnel  would  then  affect  the 
surveillance  performance  of  the  site.  Turnover  was  more  commonly  experienced  by 
personnel  assigned  to  surveillance  sites  outside  of  the  capital  (Int  1 1 ).  Since  the  rotation 
of  personnel  closer  to  the  cities  served  as  an  incentive  for  health  care  workers  and  was  a 
change  that  surveillance  personnel  in  more  remote  sites  frequently  sought  (Int  23), 
administrators  awarded  the  best  performing  staff  by  reassigning  them  to  the  more  coveted 
surveillance  sites  closer  to  cities. 

Workload 

Research  participants  noted  that  the  number  of  personnel  assigned  to  work  on 
surveillance  was  typically  insufficient  (Int  10,  Int  22,  Int  23).  Furthermore,  one  of  the 
biggest  challenges  organizations  faced  in  general  was  hiring  new  personnel  because  of 
limited  budgets  (lnt  9).  Participants  contended  that  health  care  workers  were  over-worked 
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due,  in  part,  to  frequent  assignments  of  tasks  over  and  above  the  patient  care  for  which 
they  were  responsible.  For  them,  surveillance  tasks  represented  the  addition  of  more 
activities  without  them  deriving  additional  benefits  including  salary  increases  (Int  10,  Int 
23,  Int  13).  Making  the  situation  even  more  challenging  were  the  different  types  of 
surveillance  conducted  at  any  given  site  that  were  implemented  for  different  diseases, 
thus  overloading  surveillance  staff  and  their  supervisors  (Int  3,  Int  10,  Int  15,  Int  16,  Int 
17)  with  data  collection  and  reporting  responsibilities. 

Research  participants  observed  that  when  surveillance  staff,  including  health  care 
workers,  had  many  other  activities  to  perform,  they  were  less  likely  to  collect  and  report 
surveillance  data  (Int  2).  The  participants  described  the  difference  in  the  performance 
between  a  site  with  surveillance  personnel  supported  by  an  external  organization  that 
enabled  staff  to  be  exclusively  dedicated  to  surveillance  tasks  and  a  site  without  external 
support  whose  personnel  had  multiple  tasks  to  fulfill  in  addition  to  surveillance  (Int  10). 
An  example  of  the  latter  site  was  one  affiliated  with  the  MOH.  As  might  be  expected,  the 
site  with  the  personnel  exclusively  dedicated  to  surveillance  tasks  collected  better  quality 
data  and  did  so  in  a  more  timely  way. 

Limited  surveillance  staff  skills 

Having  previous  experience  working  in  the  area  of  disease  surveillance  reduced 
staff  members’  adaptation  time  to  new  surveillance  tasks.  Research  participants  with 
experience  in  using  alternative  surveillance  system! s)  shared  the  view  that  their  prior 
experience  with  a  system  made  it  easier  for  them  to  transition  to  using  a  new  surveillance 
tool  (Int  3).  In  military  organizations  in  Peru,  most  of  the  health  care  workers  were 
unfamiliar  with  the  disease  surveillance  process  and  the  role  they  might  be  expected  to 
play  in  that  process  (Int  1 ,  Int  17). 
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When  involved  in  surveillance,  health  care  workers  had  to  incorporate  the  new 
surveillance  tasks  into  their  other  daily  tasks  and  begin  documenting  all  required  disease- 
related  information  for  their  patients  or  clients  in  a  log  book  or  other  record  (Int  4).  In 
addition,  they  had  to  learn  new  information  such  as  the  definitions  of  the  diseases  and  the 
specific  disease-related  data  or  events  that  were  being  surveyed  (Int  4).  Research 
participants  said  that  one  of  the  issues  that  health  care  workers  faced  was  a  lack  of  or 
limited  knowledge  about  epidemiology  and  disease  surveillance.  They  related  this 
limitation  to  the  education  of  health  care  professionals  in  schools  where  little  emphasis 
was  given  to  public  health  or  disease  surveillance  (Int  13,  Int  15,  Int  9). 

The  implementation  of  a  new  surveillance  tool  was  accompanied  by  a  sometimes- 
steep  learning  curve,  depending  on  the  complexity  of  the  tool.  Research  participants 
commented  that  the  likelihood  of  staff  errors  in  data  collected  and  reported  was  increased 
at  the  beginning  of  tool  use.  National  surveillance  team  members  worked  to  correct  these 
errors  by  communicating  with  staff  new  to  the  process,  often  by  telephone  (Int  17). 
However,  once  personnel  had  used  the  tool  for  about  two  months,  they  gained  facility 
with  the  tool  and  using  it  became  more  error  free  (Int  10,  Int  16,  Int  17,  Int  22).  Research 
participants  suggested  that  having  staff  with  prior  experience  in  surveillance  data 
collection  enabled  these  individuals  to  gain  expertise  in  using  a  new  surveillance  tool 
more  rapidly,  typically  within  about  one  month  (Int  3). 

Taking  into  consideration  the  challenges  of  frequent  staff  turnover,  staff 
workload,  and  limited  staff  surveillance  skills,  making  the  electronic  disease  surveillance 
task  feasible  was  recognized  by  research  participants  as  an  important  strategy  in  system 
implementation.  Surveillance  feasibility  included  designing  flexible  and  simple 
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surveillance  processes  and  creating  a  surveillance  team  at  the  site.  In  order  to  assure  a 
skilled  surveillance  workforce,  the  implementation  of  a  training  strategy  was  critical.  And 
finally,  in  order  to  increase  the  motivation  of  the  health  care  workers,  implementers 
needed  adequate  resources,  incentives,  and  support  from  direct  supervisors  at  the 
surveillance  site. 

Surveillance  Feasibility 

Study  participants  understood  that  the  effectiveness  of  a  surveillance  system 
depended  on  two  important  things  for  initiators  to  keep  in  mind:  avoiding  surveillance 
staff  work  overload  and  making  surveillance  a  feasible  task  (Int  1 1)  in  terms  of  the 
processes  and  tools  employed.  Ensuring  feasibility  involved  ensuring  simplicity, 
flexibility  and  standardization  of  both  processes  and  tools,  each  of  which  will  be 
discussed  in  the  following  section. 

Simplicity 

Surveillance  processes  needed  to  be  as  simple  as  possible  to  enable  surveillance 
staff  to  meet  expectations  and  avoid  disruption  of  their  other  assigned  activities  (Int  1 1 , 

Int  8).  Research  participants  explained  that  surveillance  systems  tended  to  fail  in  the 
presence  of  two  problems:  a)  the  system  or  its  component  processes  were  too  complex  or 
b)  administrators  or  others  held  unrealistic  expectations  for  collecting  a  great  quantity  of 
information  that  ultimately  proved  to  be  impossible  for  staff  to  manage  (Int  7).  The  latter 
problem  was  exemplified  by  one  research  participant’s  account: 

Tuvimos  la  oportunidad  [,...]de  poder  implementor  otro  sistema  de 

informacion . quisimos  hacer  lo  mismo;  nosotros  pusimos  cosas  sencillas  para 

colectar  la  informacion,  pero  el  comando  de  tumo  en  ese  momento,  queria 
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meterle  mas  information;  querla  que  ese  sistema  cargue  toda  la  information 
posible  y  eso  no  lo  hizo  viable,  a  traves  del  tiernpo  no  pudimos  implementor  eso, 
porque  se  quiso  recoger  miles  de  variables  para  que  al  final  no  se  analice  nada. 
Entonces  si  nosotros  tuvieramos  que  implementor  variables  sencillas,  creo  que 
hubiera  sido  mas  facil.  (Int  7) 

[English  translation:]  We  had  the  opportunity  [....]  to  implement  another 
surveillance  system.  but]  we  wanted  to  do  the  same  [one].  We  used  simple 
means  to  collect  the  information,  but  the  command  at  that  time  wanted  to  add 
more  variables  to  be  collected.  They  wanted  the  system  to  collect  all  the 
information  possible  and  this  vvn.s  not  viable.  We  could  not  implement  this 
because  they  wanted  to  collect  thousands  of  variables.  We  were  not  able  to 
analyze  anything  [because  the  system  did  not  work  and  did  not  collect  any 
information] .  If  we  had  implemented  simple  variables,  /  think  it  could  have  been 
easier.  (Int  7) 

The  participants  believed  that  some  aspects  of  a  disease  surveillance  system  could 
be  simplified  in  an  effort  to  contribute  to  the  potential  success  of  the  system  (Int7,  22). 
Such  simplified  efforts  could  result  from  a  planning  process  that  resulted  in  focused, 
simplified,  and  feasible  surveillance  efforts  (Int  7,  8).  As  part  of  this  effort,  strategies 
such  as  simplifying  data  collection  instruments,  reducing  report  frequency,  and 
simplifying  training  efforts  could  potentially  contribute  to  the  success  of  an  electronic 
disease  surveillance  system. 

When  determining  the  magnitude  and  type  of  surveillance  data  to  be  collected,  for 
example,  research  participants  suggested  that  consideration  be  given  to  several  issues:  the 
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need  for  the  data  and  how  data  would  be  used,  the  resources  available,  the  data  collection 


process  to  be  employed,  the  complexity  of  data  collection  forms,  the  kind  and  complexity 
of  data  analysis  (Int  8,  Int  22),  and  the  frequency  of  report  submissions  (Int  14). 
Sometimes  the  change  needed  was  reportedly  the  simplification  of  an  instrument.  Two 
research  participants  commented  that  it  had  been  necessary  for  them  to  change  a  three- 
page  form  used  by  a  surveillance  system  to  a  much  simpler  one-page  form  in  order  to 
make  it  easier  for  end  users  of  the  form  to  collect  data  (Int  22,  Int  9).  In  addition  to 
facilitating  data  collection,  simple,  user-friendly  surveillance  tools  were  noted  by  the 
participants  to  facilitate  data  reporting  for  end  users  as  well  (Int  16,  Int  8,  Int  20). 
Moreover,  two  research  participants  suggested  that  user  acceptability  could  be  enhanced 
by  having  a  simple  tool  that  users  could  quickly  load  onto  every  computer,  even  those 
with  slow  Internet  connections  (Int  8)  that,  at  the  time  of  this  study,  were  quite  common 
in  most  areas  outside  the  capital. 

Report  frequency  and  type  (individual  versus  grouped)  sometimes  posed  issues 
for  surveillance  systems  implementers  in  terms  of  the  timely  receipt  of  data.  In  one 
circumstance,  time  constraints  and  competing  tasks  necessitated  a  change  in  the 
frequency  of  grouped  report  submissions  required  of  surveillance  sites  (Int  14).  In 
another  situation,  disease  surveillance  processes  become  more  challenging  in  the  face  of 
a  disease  outbreak  or  a  disaster  since  the  incidence  of  affected  cases  increases  quickly 
and  often  substantially  (Int  8).  In  anticipation  of  this  situation,  a  national  surveillance 
team  decided  to  change  their  requirements  and  request  a  grouped  rather  than  individual 
case  report  for  future  disaster  situations  (Int  8).  Thus,  simplifying  the  reporting  schedule 
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that  could  involve  reduced  frequency  of  reporting  or  changing  from  an  individual  to  a 
grouped  report  could  yield  positive  results. 

Research  participants  were  clear  about  the  need  for  simplifying  training  resources 
in  an  effort  to  make  them  more  relevant  and  concise.  Some  research  participants  shared 
personal  experiences  associated  with  their  efforts  to  explain  surveillance  processes  to 
surveillance  staff.  In  a  training  session  recounted  by  one  research  participant,  for 
instance,  the  surveillance  team  developed  a  detailed  manual  to  guide  the  process.  Users, 
however,  found  the  manual  to  be  too  complex  and  detailed  to  be  particularly  useful. 
Implemented  learned  that  providing  a  simple  summary  of  processes  and  guidance  for 
reporting  events  was  far  more  acceptable  to  surveillance  staff  than  a  large,  highly  detailed 
manual  (Int  7). 

Standardization 

Research  participants  asserted  that  every  surveillance  process  had  to  be  “well 
mapped"  (Int  20,  Int  7).  In  general,  research  participants  in  the  study  observed  the 
importance  of  standardizing  the  processes  to  collect  and  report  data  at  every  surveillance 
site,  both  military  and  civilian,  and,  when  possible,  adapting  the  processes  to  the  tool 
used  to  report  the  data  (Int  17,  Int  22,  Int  3,  Int  4).  In  the  case  of  modifying  the 
surveillance  tool,  for  instance,  one  research  participant  shared  an  example  of  health  care 
providers  at  surveillance  sites  registering  patient  visits  by  handwriting  patient’s  names 
and  vital  signs  in  a  logbook  that  was  ultimately  used  to  extract  information  for  reporting 
(Int  14).  When  febrile  syndrome  cases  were  added  to  the  surveillance  list,  a  column  was 
added  to  the  logbook  so  health  care  providers  could  record  the  patient’s  fever  if  present 
(Int  14). 


113 


Research  participants  shared  the  intention  that  processes  be  standardized  in  every 
location  where  patients  had  initiated  contact  with  a  health  care  facility  such  as  emergency 
rooms,  ambulatory  offices,  and  hospital  admissions  (Int  22,  lnt  3,  Int  20).  However,  one 
of  the  problems  that  research  participants  observed  was  data  consistency:  not  every  site 
recorded  the  same  data  in  their  books  (Int  4).  In  some  cases,  patient  data  or  events  that 
were  required  to  be  collected  were  listed  on  a  reminder  card  as  well  as  on  the  document 
where  the  data  were  recorded.  Health  care  providers,  however,  continued  to  be  confused 
about  the  specific  data  that  they  needed  to  report  (Int  14,  Int  5,  Int  13).  For  instance, 
personnel  were  aware  of  the  need  to  report  febrile  cases  but  were  not  clear  about  the  level 
of  evidence  needed  to  determine  fever.  They  were  uncertain  if  they  needed  objective 
evidence  of  fever  or  could  accept  their  patient’s  self-report  of  fever  (Int  14)  as  reportable 
data.  They  were  also  unclear  about  the  designation  of  new  cases,  as  illustrated  by  a 
circumstance  when  a  patient  made  a  return  visit  for  the  same  complaint  (Int  18). 
Accordingly,  research  participants  recommended  that  data  collection  guidelines  clearly 
specify  the  parameters  of  data  to  be  collected  and  reported  (Int  5). 

Flexibility 

Participants  in  the  study  identified  flexibility  as  a  means  of  adaptability  that 
served  as  a  key  attribute  of  successful  electronic  disease  surveillance  systems.  They 
addressed  concerns  about  flexibility  in  several  areas:  responsiveness  to  changing 
surveillance  needs;  reporting  deadlines;  reporting  expectations;  means  of  communicating 
reports;  and  modifications  of  the  surveillance  tool.  They  also  suggested  changes  in  the 
report  accountability  process. 

Several  research  participants  shared  their  experiences  with  surveillance  systems  in 

other  countries  by  describing  the  importance  of  flexibility  not  only  when  developing  the 
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disease  surveillance  tool,  but  also  throughout  the  entire  process  of  implementation.  One 
research  participant,  for  instance,  noted  that  surveillance  actors  with  an  inflexible  vision 
of  the  system  could  jeopardize  the  implementation  process  (Int  7).  This  individual 
highlighted  the  importance  of  flexibility  from  every  person  involved  in  the  surveillance 
process  (surveillance  staff,  supervisors,  authorities,  funding  organizations)  and  during 
each  part  of  the  implementation  process  (Int  7). 

Responsiveness  to  changing  surveillance  needs 

Research  participants  underscored  the  need  for  a  disease  surveillance  system  to  be 
adaptable  to  different  epidemiological  scenarios  and  international  recommendations.  One 
research  participant  recalled  that  an  influenza  surveillance  system  proved  its  adaptability 
by  its  ability  to  incorporate  new  WHO  recommendations  when  the  system  had  already 
been  functioning  for  a  year  (Int  9).  This  participant  also  recalled  the  threat  of  pandemic 
influenza  in  2003  and  the  initiation  of  preparedness  for  it,  although  the  health  problem 
did  not  actually  occur  until  several  years  after.  The  experience  enabled  surveillance 
personnel  to  develop  several  plans  to  respond  to  future  possible  scenarios  in  terms  of 
training  and  resources  (Int  9). 

Another  research  participant  shared  the  case  of  a  laboratory  surveillance  system 
flexible  enough  for  a  surveillance  team  to  add  surveillance  for  other  health-related  events 
such  as  respiratory  viruses  and  microbial  drug  resistance  (Int  5).  Flexibility  included 
legislation  as  mentioned  in  the  context  of  a  need  to  modify  laws  or  regulations  in  order  to 
take  into  account  differing  needs  for  information.  When  the  influenza  situation  was  under 
control,  for  example,  legislation  was  modified  to  continue  to  provide  needed  information, 
but  also  to  make  data  collection  more  feasible  for  the  surveillance  staff  (Int  8,  Int  9). 
Reporting  deadlines 
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Research  participants  at  the  surveillance  staff  and  national  surveillance  team 
levels  emphasized  the  desirability  of  having  some  flexibility  in  deadline  dates  and  times 
for  surveillance  reports.  Due  to  limited  resources  at  many  sites,  study  participants  agreed 
that  reporting  deadlines  should  be  adjusted  to  reflect  what  was  feasible  at  various 
surveillance  sites.  Research  participants,  for  example,  expressed  appreciation  for  the 
change  in  report  frequency  from  twice  per  week  to  once  per  week.  They  also  appreciated 
a  change  in  the  reporting  deadline  time.  A  change  in  time  from  noon  to  midnight  (Int  14) 
on  Mondays  was  helpful  because  it  allowed  more  time  for  report  preparation  on  a  day  of 
the  week  that  participants  experienced  as  particularly  busy  (Int  15,  Int  16).  The  reasons 
given  to  justify  the  deadline  extension  included  limited  staff,  efforts  to  attend  to 
competing  tasks,  slow  Internet  connectivity,  availability  of  the  records  log(s)  used  to 
complete  the  report,  and  the  availability  of  a  means  of  communication  used  to  send  the 
report  (Int  16). 

In  reporting  expectations 

Participants  in  this  study  recognized  differences  among  surveillance  units  and 
commented  that  various  units  with  differing  levels  of  resources,  including 
communication  resources  and  expectations  for  travel,  should  not  be  held  accountable  to 
the  same  standards.  There  were  surveillance  sites  located  proximal  to  a  large  city  that  had 
all  means  of  communication  with  surveillance  staff  who  were  not  assigned  to  conduct 
surveillance  missions  external  to  the  site  (Int  4,  Int  14).  For  these  sites,  reporting  did  not 
represent  a  great  burden  when  compared  to  some  sites  that  regularly  undertook  military 
missions.  According  to  one  participant,  the  surveillance  staff  at  one  of  these  latter  sites 
would  inform  their  supervisors  that  they  were  traveling  on  a  military  mission  to  a  remote 
area  and  would  have  no  dependable  access  to  a  means  of  communication.  The  staff 
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would  then  update  the  reporting  once  they  returned  (Int  15).  Flexibility  in  reporting 
deadlines  was  typically  granted  to  units  deployed  to  areas  of  conflict  (Int  16). 
Surveillance  reporting 

Research  participants  identified  flexibility  in  the  means  of  communicating  the 
surveillance  report  as  a  key  element  that  contributed  to  the  success  of  a  surveillance 
system.  In  developing  countries,  surveillance  personnel  from  many  geographical  areas 
without  access  to  computers,  the  Internet,  or  landline  telephones  had  to  use  other  means 
of  communication  such  as  a  radio. 

Research  participants  mentioned  different  communication  modalities  they  used  to 
report  their  surveillance  data.  Depending  on  availability,  the  modalities  included  using  a 
computer  with  Internet  connectivity  or  a  landline  phone,  a  public  pay  phone,  or  a  mobile 
phone  using  a  toll  free  number  (int  12,  int  14,  int  15,  int  16).  Flexibility  in  these  modes  of 
communication  allowed  members  at  higher  levels  of  the  surveillance  system  to  reach 
surveillance  staff  in  towns  or  other  areas  where  Internet  access  was  unavailable  and 
public  pay  phones  were  distant  from  surveillance  units  (Int  16,  Int  19,  Int  21). 
Surveillance  personnel  were  able  to  transition  from  using  public  pay  phones  to  using  the 
Internet  when  Internet  access  became  available  at  their  sites.  They  found  that  reporting 
using  a  computer  was  easier  than  using  a  telephone  (int  17,  int  4).  Furthermore, 
participants  noted  that  surveillance  personnel  at  some  sites  would  use  the  radio  to 
communicate  surveillance  data  to  a  site  that  had  direct  access  to  the  system’s  electronic 
surveillance  tool.  Staff  on  the  receiving  end  of  the  radio  communication  could  then  enter 
the  information  electronically  into  the  surveillance  system  as  directed  by  the  individual 
on  the  other  end  (int  6).  A  participant  offered  the  following  comment  about 
communication  alternatives  that  have  evolved. 
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Nuestro  reporte  ha  sido  100  %  porque  buscabamos  la  forma  de  reportar  todo  lo 
que  tenia  que  ser,  por  Internet  o  por  telefono  publico,  y  se  daba  cumplimiento 
porque  dentro  de  la  base  habta  telefono  publico  si  es  que  no  habia  Internet  y  [....] 
se  ha  adicionado  lo  que  es  el  telefono  movil,  tarnbien  resulta  mucho  mas  sencillo 
reportar  lo  en  cualquier  momenta  y  en  cualquier  horario  que  podemos.  (int  14) 
[English  Translation:]  Our  report  is  100%  because  we  looked  for  the  way  to 
report  the  information,  by  Internet  or  by  public  pay  phone,  and  we  will  do  it 
because  within  the  (military)  base  there  was  a  public  pay  phone  in  case  there  was 

not  Internet  (access)  and  [ . ]  now  that  we  can  also  use  mobile  phones  to  report, 

it  is  easier  to  do  it  at  any  time.  (Int  14) 

Surveillance  tool 

Three  military-affiliated  research  participants  shared  with  the  researcher  that  the 
military-operated  disease  surveillance  system  included  a  mechatiism  to  obtain  user’s 
requests  for  surveillance  tool  modifications.  The  surveillance  representative  for  each  of 
the  Armed  Forces  could  contact  the  developers  of  the  tool  with  a  request  for 
modification.  The  surveillance  technical  assistance  team  assessed  such  requests, 
including  the  need  for  a  given  change  to  be  applied  to  one  of  the  Armed  Forces  or  across 
the  military  services  in  general.  Requests  were  assigned  a  priority  level  for  future 
development  and  implementation  (Int  12).  For  instance,  the  Armed  Forces  requested  that 
trauma  events  be  recorded  in  their  existing  surveillance  system.  These  events  were  then 
incorporated  into  the  list  of  surveillance  diseases  or  conditions  due  to  the  importance  of 
having  this  information  for  use  by  the  Armed  Forces  in  contrast  but  not  by  the  civilian 
population  (Int  4). 
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Research  participants  underscored  the  importance  of  implementing  approved 
changes  to  the  instrument  in  a  timely  manner.  Their  inability  to  see  evidence  of  timely 
changes  in  the  tool  led  them  to  believe  that  the  performance  of  the  surveillance  system 
was  compromised,  even  when  they  were  given  feedback  to  suggest  otherwise  (Int  21). 
Having  a  team  of  engineers  as  part  of  a  surveillance  team  to  assess  and  respond  to 
requests  for  changes  not  only  improved  the  flexibility  of  the  tool,  but  also  improved  the 
timeliness  with  which  the  team  responded  to  such  requests  (Int  12). 

Initiating  on-site  surveillance  teams 

Due  to  local  human  resources  constraints,  the  national  surveillance  team  learned 
that  it  was  important  to  create  a  surveillance  team  in  every  unit  (Int  9,  Int  10,  Int  13,  Int 
14,  Int  16).  They  knew  it  was  unrealistic  to  expect  that  one  person  could  be  dedicated 
exclusively  to  surveillance  (Int  15).  Therefore,  having  multiple  people  engaged  in 
surveillance  efforts  (int  16)  eased  the  overall  impact  of  the  task  and  enabled  personnel  to 
comply  more  etfectively  with  surveillance  expectations  (int  17,  int  19).  When  the  team 

worked  together,  the  system  worked  well  (Int  3).  One  person  was  assigned  responsibility 

1  h  i'  ' 

for  disease  surveillance,  but  everybody  at  the  site  collaborated  in  the  event  that  help  was 
needed  (Int  3,  Int  17,  Int  19),  especially  when  the  site  was  large  (Int  4). 

Research  participants  noted  that  some  sites  organized  the  surveillance  process  by 
assigning  one  person  to  lead  the  disease  surveillance  report  each  week  (Int  18).  They 
found  it  was  important  to  identify  and  recruit  those  health  workers  who  seemed  more 
interested  in  the  system  (Int  23)  and  who  would  be  able  to  disseminate  messages  about 
the  importance  of  the  system  (Int  14).  On  the  other  hand,  research  participants  observed 
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that  nurses  who  had  not  been  trained  in  the  system  (Int  16)  would  not  get  involved  in 
surveillance  activities  even  when  there  was  suspicion  of  a  disease  outbreak  (Int  14). 

Surveillance  site  team  research  participants  widely  recognized  training  as  one  of 
the  most  important  activities  when  implementing  a  disease  surveillance  system.  The  key 
outcome  of  disease  surveillance  was  to  have  access  to  reliable  and  timely  information  in 
order  to  make  decisions  (Int  14,  Int  1.'5,  Int  3).  Therefore,  human  resources  needed  to  be 
motivated  and  trained  in  order  to  be  able  to  effectively  and  efficiently  collect  and  transmit 
this  information  (Int  13). 

Training 

At  the  time  of  this  study,  health  care  workers  in  Peru  had  a  treatment  focus  in  the 
area  of  health  and  very  limited  training  in  public  health,  epidemiology,  or  disease 
surveillance.  Surveillance  implemented  worked  on  changing  the  focus  of  surveillance 
staff  to  preventive  medicine  where  they  would  value  preventing  the  occurrence  of 
diseases  rather  than  waiting  for  onset  of  disease  that  then  necessitated  treatment  (Int  13). 

A  participant  underscored  the  importance  of  surveillance  staff  training  in  the  following 
comment: 

Mientras  la  capacitacion  sea  mayor,  entonces  los  reportantes  tendrdn  las 
herramientas  para  un  mejor  reporte  y  para  eclucar  al  personal  de  la  importancia 
del  reporte.  (Int  16) 

[ English  translation:]  If  there  is  more  training,  then  the  surveillance  staff  will 
have  more  tools  to  do  a  better  reporting  task  and  to  educate  the  personnel  in  the 
importance  of  surveillance.  (Int  16) 
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Research  participants  suggested  that  opportunities  be  created  to  train  as  many 
personnel  as  possible  despite  the  limited  resources  available.  They  made  the  following 
suggestions: 

1 .  Offer  joint  training  across  organizations  in  order  to  achieve  reduced  training 
costs  and  aid  in  establishing  coordination  mechanisms  among  the  Armed  Forces  in  cities. 
In  that  way,  collaboration  in  responding  to  health  related  emergencies  would  be 
facilitated.  (Int  15). 

2.  Conduct  training  courses  across  all  Armed  Forces  in  cities  where  more 
surveillance  sites  were  located  (Int  15). 

3.  Reduce  training  time  to  a  one-day  course.  Address  concepts  on  surveillance 
during  a  half-day  session  and  enable  trainees  to  practice  applying  the  concepts  during  the 
remaining  half  day.  At  one  time  three-day  training  courses  were  offered  but  a  one-day 
course  afforded  trainers  two  additional  days  to  be  devoted  to  supervising  accessible  sites 
(Int  15). 

Target 

External  developers  or  individuals  from  within  the  institution  with  surveillance 
expertise  initiated  training  for  the  purpose  of  orienting  surveillance  personnel  to  new 
tools  or  systems  and  for  the  purpose  of  retraining  personnel  when  necessary.  Training 
that  ultimately  reached  all  surveillance  levels,  district,  regional  and  central,  was  most 
likely  to  yield  successful  implementation  of  a  system  (Int  20). 

According  to  research  participants,  implementing  a  new  tool  or  broader 
surveillance  system  was  typically  accompanied  by  initial  training  at  the  central  level  in 
either  the  Armed  Forces  or  the  MOH.  Once  training  of  the  central  level  team  was 
completed,  training  was  instituted  at  the  regional  level  (Int  23).  Staff  at  the  central  and 
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regional  levels  then  joined  forces  to  train  district-level  end  users.  If  an  external  group 
provided  initial  training,  this  group  made  themselves  available  for  clarification  or  support 
during  the  initial  phase  of  system  implementation  (Int  7,  Int  23). 

Context  and  content 

Trainers  had  to  consider  their  audience  when  developing  the  content  of  their 
surveillance  training  courses  (Int  20).  Health  care  workers  and  other  professionals 
differed  in  their  expectations  of  and  need  for  a  particular  depth  and  scope  of  information 
provided  and  the  language  used  in  presenting  information.  When  communicating  with 
high-level  authorities,  for  example,  the  designers  of  the  training  experience  paid  attention 
to  institutional  context  such  as  the  Armed  Forces,  where  the  highest-level  health 
authorities  did  not  have  a  medical  background. 

Instructional  designers  thus  had  to  tailor  the  training  content  and  terminology  to 
ensure  a  clear  understanding  of  disease  surveillance  by  non-health  professionals.  A 
research  participant  provided  an  additional  example  of  tailored  training  related  to  the 
increasing  reliance  of  disease  surveillance  systems  on  technology.  Adapting  the  training 
was  sometimes  necessary  to  meet  the  needs  of  senior  personnel  who  typically  had  less 
experience  with  technology  than  younger  trainees.  Their  training  sometimes  involved  a 
greater  allocation  of  time  for  learning  (Int  20). 

Delivery  Strategies 

Research  participants  described  two  primary  strategies  used  to  deliver 
surveillance  training:  a)  large-scale  training  at  one  location  which  required  attendees 
from  different  geographical  areas  of  the  country  to  travel  to  the  training  site,  and  b) 
combining  training  by  surveillance  supervisors  with  scheduled  on-site  supervisory  visits 
to  surveillance  units.  At  the  time  of  this  study,  high-level  administrators  were  considering 
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a  third  training  strategy  that  involved  using  an  on-line  environment,  but  such  training  had 
not  been  implemented  in  the  country  on  a  scale  beyond  e-mail  communication  with  staff. 
As  mentioned  earlier,  on-line  training  was  hindered  by  a  lack  of,  or  problems  with, 
consistent  Internet  access  and,  for  some  personnel,  a  lack  of  computer  access. 

Participants  observed  that  a  fundamental  problem  that  necessitated  the  training  of 
surveillance  personnel  was  a  lack  of  disease  surveillance  education  offered  to  health  care 
workers  in  their  pre-professional  preparation  programs.  Barriers  to  larger  scale,  centrally 
offered  trainings  included  a  lack  of  resources  to  fund  attendees’  travel.  In  the  case  of 
combining  training  with  supervisory  visits,  barriers  arose  with  a  lack  of  staff  time  to 
devote  to  travel,  especially  to  remote  sites,  for  supervision  as  well  as  training.  Staff 
turnover  also  posed  problems  since  basic  level  training  had  to  be  continuously  offered, 
particularly  at  more  remote  surveillance  sites. 

Surveillance  staff  training  was  mainly  conducted  at  a  central  or  regional  location, 
which  meant  surveillance  staff  members  from  all  areas  of  the  country  were  invited  to 
participate  in  these  training  courses  (Int  17).  Research  participants  described  one  of  the 
benefits  of  this  type  of  training  in  terms  of  its  importance  in  establishing  relationships 
between  and  among  system  supervisors  and  reporting  staff.  Participants  commented  that 
the  experiences  shared  during  the  training  helped  supervisors  and  surveillance  staff 
develop  closer  relationships,  which  was  essential  for  future  communication  about  the 
system.  Relationship  building  also  increased  staff  motivation  to  comply  with  disease 
reporting  guidelines  (Int  9.  Int  10.  Int  15).  In  addition,  research  participants  believed  that 
the  presence  of  a  high-level  authority  at  the  beginning  or  end  of  a  training  course 
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emphasized  to  attendees  the  importance  of  disease  surveillance.  The  presence  of  the 
authority  at  the  training  also  could  be  motivating  for  trainees  (Int  1 8). 

Bringing  surveillance  personnel  to  a  central  location  for  training  was  usually 
possible  only  with  external  funding  due  to  the  high  costs  of  travel  (Int  8,  Int  15).  Even 
with  the  availability  of  funding,  surveillance  site  personnel  often  found  it  difficult  to 
absent  themselves  from  their  site  for  any  length  of  time. 

Aside  from  the  resource-intensive  large-group  one-site  trainings  that  required 
attendees  to  travel,  training  also  was  undertaken  as  a  part  of  periodic  supervisory  visits  to 
surveillance  sites.  Supervisory  visits  to  surveillance  unit  personnel  were  resource 
intensive  due  to  limited  numbers  of  supervisors  and  expenses  incurred  with  supervisors’ 
travel  and  time.  According  to  research  participants,  the  resource  intensive  nature  of 
supervisory  visits  led  to  such  visits  serving  two  purposes:  the  provision  of  oversight  of 
personnel  and  engagement  of  site  supervisors  in  personnel  training.  Supervisors  provided 
training  or  retraining,  and  worked  with  staff  to  resolve  issues  that  hindered  adequate  site 
performance  (Int  1 5,  Int  9). 

Online  surveillance  training  was  an  additional  training  option  that  was  under 
consideration  at  the  time  of  this  study.  There  had  been  some  support  for  and  efforts 
toward  providing  online  training  to  surveillance  personnel,  though  such  training  had  not 
been  implemented  (Int  1 1 ).  Participants  expressed  the  belief  that  an  on-line  training 
option  would  be  beneficial,  particularly  as  a  means  of  addressing  resource  constraints  and 
allowing  more  flexibility  for  participants  to  attend  trainings.  Participants  explained  that 
surveillance  supervisors  used  email  and  telephone  communications  to  reinforce  the 
training  that  personnel  had  received  and  had  found  that  email  messaging,  for  example. 
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was  a  useful  way  to  clarify  staff  doubts,  send  documentation  and  information,  as  well  as 
establish  permanent  contact  with  surveillance  staff  being  supervised  (Int  15).  One 
problem  however,  was  the  continuing  high  rate  of  staff  turnover  at  surveillance  sites,  thus 
presenting  a  continuous  need  for  basic  training. 

Type 

Research  participants  described  two  considerations  in  delivering  training  designed 
to  meet  the  needs  of  surveillance  staff.  Training  had  to  be  maintained  continuously 
whether  the  system  was  in  the  initial  stage,  expanding,  or  consolidating  the  system,  and 
performed  in  a  comprehensive  way.  These  two  considerations  will  be  discussed  in  the 
following  section. 

Continuous  training.  A  system  cannot  survive  without  continuous  training 
especially  considering  high  staff  turnover  (Inti ).  This  turnover  was  commonly  seen  in  the 
Armed  Forces  where  personnel  were  constantly  relocated  around  the  country  and  at  the 
MOH  where  employees  were  constantly  seeking  opportunities  to  be  assigned  to  sites 
closer  to  large  cities  (Int  1,  Int  9,  Int  16,  Int  23). 

Research  participants  agreed  that  training  should  not  only  be  offered  on  an  on¬ 
going  basis  but  also  be  conducted  as  frequently  as  possible  (Int  16.  Int  18).  Research 
participants  from  the  region  with  the  most  successful  disease  surveillance  network  stated 
that  the  epidemiology  team  provided  monthly  training  for  the  surveillance  staff  at  the 
district  level  that  focused  on  various  surveillance-related  topics.  They  identified  frequent 
on-going  staff  training  as  one  of  the  most  important  contributing  factors  to  their 
exceptionally  good  performance  in  disease  surveillance  (Int  23). 

Comprehensive  training.  A  research  participant  believed  that  the  format  and 

content  of  the  training  course  was  essential  to  the  success  of  several  systems  with  which 
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he  had  worked  (Int  7).  His  recommendation  for  the  initial  phase  of  the  evolution  of  the 
system  was  to  have  training  courses  no  longer  than  two  days  duration.  Longer  courses 
appeared  to  overwhelm  health  care  workers  and  made  it  difficult  for  them  to  retain  the 
concepts  needed  to  start  reporting  (Int  7).  The  objective  of  this  methodology  was  to  give 
the  end  users  information  that  could  be  immediately  used  when  they  returned  to  their 
sites  (Int  7).  According  to  research  participants,  surveillance  staff  should  start  this  new 
task  immediately  after  they  receive  training  (Int  1 8,  Int  7).  A  participant  shared  his 
experience  with  one  system  that  was  not  ready  to  be  used  though  training  was  received. 
He  explained  that  end  users  had  lost  their  motivation  by  the  time  the  system  was  ready  to 
be  used,  so  their  performance  was  quite  low.  Reporting  staff  was  overwhelmed  by  their 
other  multiple  tasks  (Int  8). 

According  to  research  participants,  the  content  addressed  in  training  should,  at 
minimum,  include  the  following:  a)  the  purpose  and  importance  of  disease  surveillance, 
b)  the  role  of  health  care  workers  in  surveillance,  c)  the  data  collection  and  disease 
reporting  process,  d)  data  analysis  and  interpretation,  and  e)  the  detection  of  and 
appropriate  response(s)  to  disease  outbreaks  or  events  (Int  1 ,  Int  7,  Int  8).  Each  of  these 
content  areas  will  be  further  explained  in  the  following  section. 

Training  should  include  content  on  concepts  related  to  surveillance  and  functions 
that  disease  surveillance  systems  serve,  the  legislation  that  mandates  and  guides 
surveillance,  and  information  about  the  most  prevalent  and  relevant  diseases  in  their 
community,  region,  country,  and  internationally.  Trainers  should  explain  how  the  disease 
surveillance  information  collected  will  be  used  for  making  public  health  decisions  in 
public  health  (Int  7), 
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Surveillance  staff  should  be  taught  who  and  how  to  notify  supervisors  about 
disease  events  of  concern  and  which  events  required  notification.  The  reporting  process 
had  to  be  clearly  explained,  with  time  assigned  to  allow  trainees  to  practice  using  the  data 
collection  tool  and  reporting  system  (Int  7).  A  research  participant  recommended  that 
examples  of  situations  (e.g.  how  to  report  a  patient  being  seen  by  a  health  care  provider 
for  the  second  time  with  the  same  complaint)  experienced  by  the  surveillance  staff  should 
be  documented  and  shared  with  others  (Int  18).  Clearly  defined  terms  were  essential; 
especially  what  constituted  a  reportable  and  notifiable  disease  event.  Participants 
commented  that  some  staff  would  notify  the  team  about  events  not  required  to  be 
reported.  Participants  also  recommended  that  definitions  should  also  be  shared  with  the 
physicians  who  made  the  diagnosis  of  such  events.  When  physicians  did  not  consider 
such  definitions  when  writing  the  diagnosis,  it  was  difficult  for  the  surveillance  staff  to 
find  complete  information  to  report  the  event  (Int  14). 

It  was  recommended  that  training  provide  surveillance  staff  with  skills  to 
understand  and  interpret  the  data.  Post-training  evaluations  revealed  that  surveillance 
staff,  especially  those  located  at  sites  outside  the  capital,  did  not  know  the  basic  concepts 
of  epidemiology,  disease  surveillance,  and  outbreak  detection  and  response.  Therefore, 
they  were  not  prepared  to  respond  adequately  to  the  information  collected  (Int  1 1). 
National  surveillance  teams  and  surveillance  staff  at  the  site  specifically  emphasized  the 
need  for  training  in  epidemiology  measurements  and  basic  statistics  for  better 
interpretation  of  the  data  collected. 

Surveillance  staff  needed  training  that  enabled  them  to  identify  disease  outbreaks 
and  other  events  and  appropriately  respond  to  them.  Research  participants  requested 


127 


training  on  diagnosing  and  treating  the  health  events  commonly  presenting  at  their  sites 
(Int  18).  Furthermore,  they  needed  explanations  about  strategies  to  detect  and  respond  to 
outbreaks.  Having  this  information  empowered  staff  and  reinforced  the  importance  of 
their  disease  surveillance  task  (Int  16).  Research  participants  shared  their  regrets  about 
using  a  training  approach  that  focused  solely  on  the  process  of  reporting  (Int  1 1). 

Train  the  trainers  approach 

Research  participants  recommended  a  simple  and  replicable  training  approach 
that  enabled  those  who  had  been  trained  to  train  personnel  at  their  sites  (Int  18).  Trainees 
then  became  trainers,  thus  contributing  to  a  more  rapid  spread  of  knowledge  about 
electronic  disease  surveillance  (Int  6).  The  provided  examples  of  such  an  approach  that 
included  nurses  replicating  the  training  for  military  personnel  before  these  personnel 
deployed  to  remote  areas  in  the  rainforest  (Int  19).  In  one  organization,  personnel  agreed 
that  every  person  who  was  trained  had  to  replicate  the  training  for  the  rest  of  the  team  (Int 
4).  The  creation  of  regional  teams  in  charge  of  the  training  was  a  cost  reduction  measure 
since  the  national  team  would  not  have  to  be  responsible  of  all  training  throughout  the 
country  (Int  10).  A  train-the-trainers  strategy  made  training  more  frequently  offered  and 
more  widely  disseminated  across  the  country  (Int  8,  Int  9). 

Institutionalization  of  training 

Research  participants  mentioned  that  a  system-wide  training  allowed  them  to 
develop  an  annual  plan  for  training  in  epidemiology  and  disease  surveillance  for  their 
organization  for  the  first  time.  They  began  to  incorporate  this  training  into  the  curriculum 
of  military  nurses  before  they  graduated  (Int  2)  and  updated  courses  military  health 
personnel  received.  This  training  was  mandatory  and  part  of  the  surveillance  personnel 
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training  curriculum  (Int  21 ).  The  regional  epidemiology  team  developed  a  certificate  of 
epidemiology,  which  was  a  six-month  online  training  program  for  surveillance  staff  at  the 
regional  and  local  level.  This  training  represented  a  unique  opportunity  for  certificate 
holders  to  pursue  a  graduate  degree  in  an  area  of  their  expertise  (Int  23). 

In  summary,  training  was  greatly  identified  as  a  need  for  the  successful 
implementation  of  these  systems  and  as  a  way  to  address  challenges  such  as  staff 
turnover,  workload  and  limited  skills  in  surveillance.  These  challenges  were  commonly 
described  across  stakeholders'  groups  interviewed.  The  training  strategy  in  place  should 
be  continuous  and  comprehensive,  with  a  use  of  the  train-the-trainers  approach  to 
increase  the  coverage  of  the  population  trained.  Finally,  in  order  to  be  sustainable,  this 
training  has  to  be  part  of  the  curricula  offered  by  the  institution  or  organization. 

The  skills  acquired  through  training  were  a  requisite  for  the  reporting  staff  to 
fulfill  the  surveillance  task.  However,  training  also  showed  to  increase  the  motivation  of 
these  health  care  workers.  Motivation  was  important  to  incentivize  these  workers  as  they 
were  assigned  to  one  more  task  when  they  were  already  overworked  and  received  low 
wages.  The  following  section  will  explore  the  different  strategies  put  in  place  to  increase 
motivation  of  the  reporting  staff. 

Surveillance  Staff  Motivation 

Research  participants  contended  that  any  surveillance  system  needed  motivated 
and  committed  surveillance  personnel  in  order  to  be  successful  (Int  8,  Int  9).  This 
contention  was  reflected  in  the  following  comment  by  a  participant: 

“Todo  si  sterna  debe  empezar  con  el  compromiso  del  personal  reportante,  si  no 
hay  esu,  aun  cuando  el  sistema  fuera  perfecto,  no  funciona  (Int  8) 


129 


[English  Translation:]  "Every  system  should  start  with  the  commitment  of  the 
reporting  staff;  if  that  is  not  present ,  even  when  the  system  is  perfect,  it  won 't 
work”.  (Int  8) 

They  observed  that  low  performing  sites  lacked  motivated  staff,  especially  in  sites 
outside  of  the  capital  (Int  13).  Life  outside  the  capital  is  slower  and  people  in  general 
have  less  drive  to  perform  well  in  comparison  with  the  capital  city.  On  the  other  hand, 
they  saw  that  highly  motivated,  committed  personnel  more  effectively  performed  their 
disease  surveillance  tasks  (Int  15).  Research  participants  mentioned  people  at  the 
organizations  where  they  worked  who  were  already  motivated  to  do  a  good  job  as  part  of 
their  personal  work  ethic.  One  of  the  strategies  implemented  used  was  to  identify  these 
individuals  and  assign  them  surveillance  duties.  However,  this  was  not  always  the  case 
with  all  the  surveillance  sites  across  the  country  (Int  14,  Int  17,  Int  22). 

As  previously  mentioned,  research  participants’  observed  that  awareness  of  the 
health  care  workers  about  the  importance  of  epidemiology  and  disease  surveillance  in  the 
country  was  quite  low,  even  at  the  MOH  (Int  17,  Int  18,  Int  22).  They  emphasized  the 
importance  of  heightening  awareness  within  the  entire  organization  when  implementing  a 
disease  surveillance  system,  and  suggested  beginning  at  the  surveillance  staff  level. 
Surveillance  A  wareness 

Enabling  staff  to  increase  their  awareness  of  the  purpose  and  importance  of 
surveillance  and  specifically  the  significance  of  surveillance  staff  roles  and  tasks  (Int  10, 
Int  14,  Int  21 ,  Int  22,  Int  23,  Int  8)  was  an  outstanding  need.  The  surveillance  staff 
needed  to  be  educated  about  disease  surveillance  in  order  to  understand  the  importance  of 
their  surveillance  tasks  and  the  critical  role  they  and  the  data  they  collected  ultimately 
played  in  disease  prevention  (Int  13,  Int  14).  For  instance,  surveillance  staff  had  to 
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register  cases  and  collect  flu  samples  as  strategies  critical  to  the  annual  WHO  vaccine 
development  process  (Int  10).  Research  participants  noted  that  once  surveillance  staff 
understood  the  consequences  of  submitting  low  quality  or  missing  data  or  not  forwarding 
it  in  a  timely  manner  (Int  1 1,  Int  14,  Int  15),  staff  performance  improved  (Int  16). 

Research  participants  recommended  that  implementers  of  disease  surveillance 
systems  work  to  change  the  views  of  those  surveillance  staff,  particularly  those  who  saw 
their  task  as  merely  routine  or  inconsequential  (Int  16,  Int  21.  Int  22).  They  suggested 
reiterating  messages  about  the  important  role  that  surveillance  staff  played  in  detecting 
and  averting  disease  events  with  epidemic  potential  (Int  19).  Feedback  through  the 
publication  and  dissemination  of  surveillance  bulletins  reinforced  this  message  (Int  21). 
Staff  was  also  very  motivated  when  they  were  able  to  detect  outbreaks  at  an  early  stage 
and  prevent  a  more  damaging  situation  (Int  17,  Int  19).  They  found  it  motivating  to  apply 
their  knowledge  as  in  the  creation  of  graphs  using  case  data  to  assess  disease  levels  for 
their  target  population  (Int  2).  One  participant  observed  the  difference  between  staff: 

Hay  algunos  que  reportan  por  obligation ,  y  no  hacienda  la  estadistica  adecuada , 
solo  por  cumplir,  estos  no  estan  concientizados.  Pero  hay  otros  que  trabajan  con 
600  personas  en  submaritios,  y  viendo  los  cuadros  como  van,  los  picos,  y  manejar 
la  situation.  (Int  16) 

[English  Translation:]  There  are  some  who  report  only  because  they  feel  they 
need  to;  they  do  not  analyze  their  information,  only  to  comply  with  this  task,  those 
are  not  aware  of  the  importance  (of  the  system).  But  there  are  others  that  work 
with  600  people  in  a  submarine  and  they  look  at  what  their  charts  show,  the  peaks 
that  can  present  to  manage  this  situation.  (Int  16) 
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While  awareness  of  surveillance  staff  was  critical  for  engagement  on  the  reporting 
tasks,  implemented  soon  realized  that  all  health  care  workers  had  to  be  educated  about 
the  system,  the  importance  of  surveillance,  and  their  role  in  the  documentation  of  disease 
(Int  14).  All  health  care  professionals  needed  to  be  trained  in  surveillance-related  data 
collection.  Physicians  at  disease  surveillance  sites,  for  instance,  needed  to  know  how  to 
accurately  record  the  symptomatology  and  diagnosis  of  the  patients  they  interviewed  and 
treated  (Int  22). 

One  strategy  used  to  inform  all  professionals  about  disease  surveillance  was  to 
regularly  address  findings  in  meetings  with  leaders.  Research  participants  mentioned  that 
the  head  of  one  surveillance  site  held  monthly  meetings  involving  all  department  chiefs 
to  update  them  on  the  diseases  they  surveyed  and  inform  them  about  current 
epidemiological  data  of  concern  (referred  to  as  “epidemiological  alerts”).  High  level 
authorities  also  used  their  media  departments  to  disseminate  information  by  e-mail  or 
other  means  to  all  personnel  in  the  organization  about  the  current  epidemiological  status 
of  the  disease(s)  they  surveyed  (Int  22).  Research  participants  observed  that  when 
information  about  a  reportable  disease  was  highly  disseminated,  physicians  were  more 
likely  to  report  required  information  about  the  disease  (Int  22),  thus  facilitating  the 
surveillance  task  for  everyone. 

Research  participants  explained  that  military  organizations  were  hesitant  to  report 
disease  outbreaks  because  personnel  thought  they  would  be  punished  or  blamed  for  the 
incident  (Int  11,  Int  16,  Int  18).  Moreover,  military  personnel  who  were  usually  non¬ 
health  care  professionals  did  not  understand  the  importance  of  reportable  diseases  (Int 
16).  Surveillance  staff  mentioned  that  they  experienced  a  lack  of  interest  and  support 
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from  their  direct  supervisors  as  evidenced  by  their  request  that  they  deprioritize  data 
reporting  in  deference  to  engaging  in  other  tasks  or  assignments  unrelated  to  surveillance 
(Int  16).  At  many  sites,  commanders  and  others  perceived  disease  surveillance  as  a  last 
priority  (Int  16). 

Research  participants  recognized  the  need  for  increasing  the  awareness  of  non¬ 
health  military  personnel  about  the  importance  of  disease  surveillance  systems  and  the 
contributions  such  systems  can  make  to  their  mission,  the  lives  of  their  soldiers,  and  the 
citizens  and  country  they  serve.  These  personnel  should  be  educated  using  simple 
language  (Int  14,  Int  15,  Int  16).  Their  understanding  of  the  critical  nature  of  disease- 
related  reporting  may  facilitate  the  reporting  process  within  their  own  sites  (Int  14). 
Participants  suggested  applying  the  data  already  collected  at  their  sites  to  illustrate  the 
value  of  the  information  they  collected.  One  participant  proposed,  as  an  example,  that 
presenting  the  incidence  of  respiratory  infections  while  ships  were  in  dock  compared  to 
the  increased  incidence  of  such  infections  after  the  ships  got  underway,  would  be  of  great 
value  to  explain  the  increased  risk  of  transmission  of  respiratory  infections  to  their 
commanders  (Int  14). 

The  importance  of  disease  surveillance  accorded  by  administrators  of  an 
organization  reinforced  the  importance  perceived  by  staff  at  the  organization  (Int  17). 
Research  participants  observed  that  when  the  highest  ranked  person  at  the  site  knew 
about  the  system  and  the  need  to  comply  with  it,  the  chance  that  the  task  would  be 
accomplished  by  the  organization  was  heightened  (Int  17,  Int  18,  Int  21 ).  A  participant 
noted  that  having  an  Armed  Forces  commander  who  was  aware  of  the  importance  of 
disease  surveillance  systems  led  that  individual  to  support  the  surveillance  staff  in  the 
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tasks  they  needed  to  accomplish  (Int  16).  In  contrast,  the  participant  recalled  a 
circumstance  where  an  uninformed  Armed  Forces  base  commander  called  in  surveillance 
staff  to  castigate  them  for  reporting  a  disease  outbreak.  Once  the  medical  director 
explained  their  actions  and  the  surveillance  system  to  the  commander,  the  outcome  was 
positive  in  that  measures  were  instituted  to  prevent  similar  misunderstandings  in  the 
future  (Int  16). 

Research  participants  stated  that  the  dissemination  of  information  about  the 
existence  and  products  of  a  disease  surveillance  system  was  a  key  component  to  create 
this  awareness  of  the  importance  of  the  system  (Int  14).  However,  the  dissemination 
process  could  also  be  quite  challenging  (Int  15,  Int  16).  A  lack  of  information 
dissemination  about  the  surveillance  system  invariably  resulted  in  an  organization-wide 
lack  of  awareness  about  surveillance.  Participants  commented  that  when  they  supervised 
sites,  they  could  see  evidence  that  the  head  of  the  site  did  not  view  the  system  as 
important  and  that  the  commander  and  personnel  were  uninvolved  in  surveillance.  Non¬ 
health  care  military  personnel  did  not  regard  surveillance  as  part  of  their  responsibility, 
viewing  it  instead  as  the  sole  responsibility  of  health  care  workers  (Int  2).  In  order  to 
remedy  a  situation  where  personnel  in  general  lucked  awareness  about  the  existence  and 
purpose  of  their  organization-based  disease  surveillance  system,  research  participants 
strongly  recommended  disseminating  information  about  the  system  through  regular 
channels  (Int  17).  They  observed  that  the  process  of  dissemination  took  time  because  of 
existing  bureaucratic  barriers  to  sending  formal  documentation  to  every  reporting  unit 
(Int  15,  Int  16).  However,  the  strategy  of  broad  dissemination  of  information  to  the  entire 
organization  was  important.  It  not  only  informed  personnel  about  a  system  integral  to  the 
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organization,  but  also  gave  the  commander  the  opportunity  to  become  involved,  share  her 
or  his  perspectives,  and  ultimately  support  surveillance  personnel  in  the  tasks  they  had  to 
perform  (Int  15). 

Surveillance  Site  Resources 

Motivated  staff  typically  had  the  resources  required  to  complete  their  surveillance 
tasks  such  as  access  to  a  computer,  a  telephone,  or  a  community-based  Internet  cafe  (Int 
14,  Int  19,  Int  9).  Participants  found  that  providing  these  resources  such  as  the  purchase 
of  computers  for  their  work  group  was  quite  motivating  (Int  8). 

Particularmente  yo  no  espero  que  me  den  algun  reconocimiento  simpleniente  to 
tomo  como  una  parte  importante  de  mi  trabajo  y  como  un  granito  de  arena,  el 
sistema  finalrnente  como  dijo  el  doctor,  somos  128  reportantes,  entonces  yo  soy 
uno  de  los  128  pero  generalmente  no  esperamos  ningi'm  reconocimiento  o  algo, 
quizd  de  repente  un  poco  de  apoyo  en  lo  que  se  refiera  a  tecnologia  ,  por  ejemplo 
en  este  caso,  como  te  decia,  a  veces  los  comandos  no  quieren  dar  dos  soles,  o  sea, 
dos  soles  que  no  es  nada  pero  inclusive  esos  dos  soles  no  les  dahan  entonces  el 
enfermero  decia,  ‘Si  no  yo  tengo  que  sacar  de  mi  dinero  para  reportar’,  entonces 
tampoco  no  era  justo.  ( Int  14) 

l English  Translation:]  Personally,  /  do  not  expect  any  recognition.  I  simply  take 
it  as  an  important  part  of  my  work,  as  my  small  contribution.  The  system  is 
finally,  like  the  doctor  said,  we  are  128  reporting  staff,  then  I  am  only  one  of 
those  128  so  we  do  not  expect  any  recognition.  Maybe  just  support  with  access  to 
the  technology.  For  example,  I  was  telling  you  about  that  time  when  the 
commander  did  not  want  to  give  the  nurse  two  soles  to  do  the  reporting  (to  use  the 
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Internet  cafe).  It  is  not  fair  then  the  nurse  has  to  spend  his  own  money  to  do  this 

(Int  14). 

During  the  planning  phase,  participants  described  how  important  it  was  to 
consider  the  resources  required  to  implement  a  system.  The  economic  resources,  human 
resources  and  the  infrastructure  already  in  place  had  to  be  assessed  in  order  to  determine 
needs  at  the  central,  regional  and  district  levels  (Int  1,  Int  21).  The  resource  planning 
should  be  based  on  the  system  and  technology  being  implemented,  including  laying  out 
all  the  processes  such  as  the  data  collection,  data  reporting  and  analysis  of  the  data 
undertaken  through  the  evolution  of  the  system,  including  the  initial,  expansion,  and 
consolidation  phases  (Int  15). 

Research  participants  emphasized  the  importance  of  considering  the  resources 
available  when  deciding  to  expand  the  system  (Int  1).  One  problem  to  avoid  was  the 
funding  of  recurring  payments  because  if  the  payment  failed,  then  the  system  failed.  For 
instance,  several  surveillance  sites  paid  for  Internet  accessibility.  However, 
administrative  problems  could  slow  this  payment  and  interrupt  the  service,  thus  bringing 
surveillance  reporting  to  a  halt  (Int  8,  Int  9). 

Availability 

Resources  were  one  of  the  biggest  challenges  that  the  implementation  of 
surveillance  systems  faced.  Research  participants  expressed  great  concern  regarding  the 
resources  that  their  organizations  or  institutions  had  in  place,  the  resources  that  were 
committed  to  surveillance,  and  the  external  funding  available.  Participants  identified 
three  issues:  limited  human  resources,  lack  of  appropriate  equipment,  and  lack  of  access 
to  technology  and  the  Internet  as  the  most  common  barriers  to  implementation. 
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Lack  of  access  to  Internet  connectivity  was  one  of  the  main  concerns  among 
research  participants.  There  was  controversy  among  implemented  on  developing  online 
vs.  standalone  systems  because  many  sites  did  not  have  access  to  the  Internet.  Several 
sites  were  not  incorporated  into  the  main  surveillance  system  as  they  did  not  use 
computers  and  using  the  phone  for  reporting  would  only  cause  a  work  overload  for  them 
(Int  6). 

The  main  reasons  cited  for  not  having  Internet  access  were  a  “lack  of  funding” 
and  a  “lack  of  availability  of  such  services  in  their  geographic  area.”  (Int  8,  Int  13,  Int  14, 
Int  15,  Int  1 8,  Int  19,  Int  4,  Int  6).  For  military  organizations,  some  units  had  to  go  on 
missions  for  several  weeks  to  areas  with  no  access  to  computers  or  phones  (Int  4,  Int  14). 
In  other  cases,  participants  mentioned  that  surveillance  staff  had  to  use  Internet  cafes  or 
their  own  personal  computers  or  telephones  to  make  the  report  (Int  10,  Int  15,  Int  17,  Int 
21).  Internet  accessibility  was  also  a  challenge  for  training  sessions  (Int  6). 

Unfortunately,  even  when  some  sites  had  Internet  access,  personnel  in  surveillance 
offices  did  not  share  that  access  (Int  14).  Internet  access  in  certain  areas  of  the  country 
was  often  very  slow,  making  the  reporting  task  even  harder  (Int  16). 

Participants  noted  that  a  lack  of  adequate  equipment  was  also  a  frequent  problem 
(Int  23,  Int  9).  Commonly,  there  were  only  a  few  computers  at  the  site  causing 
surveillance  staff  to  wait  for  an  available  computer  in  order  to  input  or  report  their  data 
(Int  16).  Computer  break  down  (Int  16)  and  public  pay  phones  that  were  out  of 
commission  exemplified  additional  equipment  problems  (Int  18,  Int  19).  At  the  regional 
level,  the  epidemiology  office  did  not  manage  data  with  their  own  server,  thus  having  to 
use  the  regional  government  network  which  was  very  slow  since  it  served  many  work 
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areas  (Int  23).  Additionally,  research  participants  commented  about  the  inaccessibility  of 
software  resources  needed  to  use  the  analytical  tools  they  needed  (Int  8). 

Research  participants  commented  that  resources  could  not  be  neglected  at  the 
surveillance  supervisors'  team  level  as  well.  Supervisors  needed  an  adequate  work 
environment  and  access  to  tools  to  monitor  the  system  and  provide  support  to  the 
surveillance  staff.  A  fully  equipped  office  with  computers,  Internet  access,  and  telephone 
lines  to  communicate  with  the  surveillance  staff  were  essential  in  the  process  of  disease 
surveillance  (Int  21). 

Administrative  support 

Beyond  the  lack  of  resources,  research  participants  also  observed  that  their 
commanders  did  not  prioritize  the  surveillance  tasks  even  when  they  were  mandatory. 
Even  when  equipment  was  purchased,  it  was  assigned  to  the  highest  ranked  personnel 
instead  of  the  surveillance  staff  at  the  site  (Int  14).  Many  times,  direct  supervisors  at  the 
site  would  not  allow  the  surveillance  staff  access  to  computers  or  phones  already  in  place 
(Int  14,  Int  15,  Int  16,  Int  4).  Furthermore,  surveillance  staff  would  be  asked  to  do  other 
tasks  like  administrative  activities  instead  of  prioritizing  their  surveillance  role  (Int  16). 
Research  participants  mentioned  that  they  often  would  not  receive  logistics  support  to  do 
surveillance,  such  as  the  cost  for  using  an  Internet  cafe  when  they  needed  to  send  a  report 
or  even  the  permission  required  to  leave  the  base  to  access  the  Internet  (Int  16,  Int  19). 

Performance-based  access 

Resources  had  to  be  present  at  every  level.  An  initial  strategy  was  to  make  sure 
the  users  would  have  the  resources  to  be  able  to  use  the  electronic  tool  developed  (Int  I, 
Int  8).  As  one  participant  commented; 
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Si  vas  a  introducir  una  herramienta  que  utiliza  tecnologias  web,  las  personas,  al 
menos  clebe  de  haber  una  computadora  para  que  las  personas  lo  hagan,  algo  tan 
simple  como  eso.  (Int  1) 

If  you  are  going  to  introduce  a  tool  that  uses  web-based  technologies,  the  people, 
at  least  need  to  have  a  computer  so  they  can  use  the  tool,  something  as  simple  as 
that.  (Int  I) 

In  the  case  of  the  system  used  by  the  Armed  Forces,  information  could  be  sent  by 
calling  a  toll  free  number,  thus  requiring  access  to  a  landline  or  cellular  telephone  (Int  I). 
Research  participants  emphasized  that  surveillance  sites  with  access  to  technology  used 
by  the  system  had  few  problems  with  reporting  (Int  14).  Furthermore,  having  a  usable 
Internet  connection,  especially  one  that  was  not  slow',  helped  with  their  surveillance  tasks 
(Int  9).  Having  a  computer  with  Internet  access  also  meant  surveillance  staff  could  view 
their  reporting  data,  analyze  the  information,  and  understand  the  status  of  the  diseases 
surveyed  within  their  population  (Int  19,  Int  23,  Int  3). 

Existing  resources  use 

In  a  resource  limited  setting,  implementers  should  rely  as  much  as  possible  on  the 
resources  in  place  as  this  would  help  the  sustainability  of  the  system  (Int  21). 
Implementers  considered  Internet  access  when  selecting  the  sites  for  implementing  the 
sentinel  surveillance  system  (Int  8,  Int  15).  At  the  same  time,  they  paid  attention  to 
country  context,  which  sometimes  provided  favorable  opportunities  for  the 
implementation  process.  A  recently  adopted  approach  by  the  MOH  to  allocate  budgets 
according  to  surveillance  performance  results  enabled  implementers  to  request  funding  to 
strengthen  the  MOH  surveillance  system  (Int  10).  With  this  money,  they  requested  the 
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purchase  of  equipment,  information  systems,  human  resources,  and  the  logistics  required 
(Int  10). 

In  the  same  way,  surveillance  administrators  creatively  solved  resource 
challenges.  They  hired  temporary  workers  to  start  monitoring  the  system.  Personnel  at 
sites  without  Internet  and  telephone  were  trained  to  communicate  surveillance  data  by 
radio  to  sites  that  had  a  means  of  reporting  the  information  (Int  8).  When  no  budget 
existed  to  buy  computers,  implementers  lobbied  at  the  regional  surveillance  level  for 
support  that  would  allow  the  use  of  regional  resources  for  surveillance  tasks  (Int  10). 
Research  participants  mentioned  that  it  was  essential  to  create  awareness  about  the 
importance  of  surveillance  at  all  organizational  levels.  This  awareness  not  only  allowed 
for  assistance  with  resources  but  also  facilitated  the  processes  associated  with 
surveillance,  such  as  a  Director’s  willingness  to  sign  a  document  in  order  to  have  a 
sample  leave  the  site  (Int  22). 

Incentives 

Implementers  observed  a  positive  impact  of  non-financial  incentives  on  the 
morale  of  the  surveillance  staff.  For  instance,  a  national  “Epidemiology  Day”  was 
designated  to  increase  awareness  of  disease  surveillance  and  to  honor  those  who 
demonstrated  excellent  performance.  Administrators  at  the  national  surveillance  level 
created  a  ranking  to  identify  the  best  performing  units  during  the  training  courses  (Int 
18).  Surveillance  supervisors  used  this  ranking  to  congratulate  their  staff  through  a 
formal  document  and  during  a  once  a  year  ceremony  in  association  with  “Epidemiology 
Day”  that  was  held  at  the  organization  (Int  21,  Int  3,  Int  4.  Int  15,  Int  19).  Research 
participants  believed  that  such  performance  site  comparisons  contributed  to  staff 


140 


motivation  (int  23).  As  an  additional  incentive,  personnel  from  the  best  performing  sites 
across  the  country  were  invited  to  participate  in  the  “Epidemiology  Day”  celebration  and 
had  their  travel  funded  (Int  4).  National  level  administrators  prepared  training  diplomas 
for  that  day  that  were  given  to  recipients  by  the  highest  authority  in  their  organizations 
(Int  23). 

As  mentioned  earlier,  training  was  perceived  as  an  incentive  for  surveillance  staff, 
especially  for  those  who  worked  in  sites  outside  the  capital  city  (Int  4,  Int  9,  Int  1 8). 
Research  participants  indicated  that  personnel  highly  valued  the  training  courses  that 
funded  their  travel  expenses  to  attend  the  course  (Int  8).  They  also  held  training  courses 
that  offered  medical  education  credits  which  also  proved  to  be  an  incentive  for  attendees 
(Int  2). 


Training  as  a  motivator 

Besides  the  benefits  already  mentioned,  training  was  perceived  as  a  motivator  (Int 
9).  Surveillance  staff  who  participated  in  training  gained  more  tools  with  which  they 
could  improve  their  performance  and  the  knowledge  to  educate  all  personnel  on  the 
importance  of  disease  surveillance  (Int  16).  The  use  of  an  electronic  disease  surveillance 
system  enabled  surveillance  staff  to  have  knowledge  about  the  overall  health  situation  at 
their  sites  as  well  as  the  region  and  country,  which  they  also  considered  a  motivating 
experience  (Int  17). 

Finally,  as  a  consequence  of  the  training  process  research  participants  also 
identified  areas  where  system  improvements  were  needed.  They  identified  the  need  for 
public  health  training  to  be  strengthened  following  undergraduate  school  for  all  health 
care  workers  (Int  10,  Int  15).  They  discovered,  for  example,  that  more  training  in  health 
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informatics  should  be  developed  and  offered  since,  at  the  time  of  this  study,  only  one 
school  in  the  country  offered  a  certificate  and  a  master’s  degree  in  health  informatics  (Int 
20).  They  also  advocated  that  training  opportunities  in  epidemiology  offered  by  the  MOH 
be  opened  to  health  care  workers  from  all  institutions  including  the  Armed  Forces  (Int  4). 

Accountability 

In  the  case  of  delays  in  surveillance  report  submissions,  most  military 
organizations  addressed  accountability  by  means  of  written  formal  documentation  of  the 
incident.  Documentation  was  usually  followed  by  sanctioning  the  individual(s) 
responsible.  A  sanction  might  mean  the  individual's  supervisors )  personally  addressed 
the  issue  with  them  or  their  supervisors  wrote  an  unsatisfactory  performance  note  on  their 
resumes.  Military-affiliated  research  participants  recommended  a  more  flexible  approach 
to  such  problems.  They  suggested  direct  contact  with  surveillance  staff  be  made  as  an 
initial  effort  to  assess  the  reason  for  the  delay  and,  if  reasonable,  to  provide  a  deadline 
extension.  In  the  case  of  a  continued  delay,  research  participants  recommended 
immediate  notification  of  the  superior  of  the  surveillance  staff.  Additional  delays  would 
result  in  informing  the  health  department  chief  and  finally  the  unit  commander  (Int  15). 
Participants  shared  that  the  current  military  process  of  documenting  and  sanctioning  hurt 
their  morale.  It  resulted  in  site  surveillance  staff  compliance  with  reporting  deadlines, 
but  only  for  the  purpose  of  avoiding  being  sanctioned  (Int  14,  Int  16). 

In  summary,  in  order  to  generate  timely  and  quality  data,  the  reporting  staff 
needed  to  be  skilled  and  motivated;  have  access  to  adequate  resources  to  collect  and 
transmit  the  data  collected  and  be  assigned  to  a  feasible  task.  This  feasibility  comprised 
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having  simple,  flexible  and  standardized  procedures  in  place,  including  the  surveillance 
tool  used  by  the  system. 

Next,  the  researcher  will  describe  the  role  and  functions  of  the  national  and 
regional  team  to  provide  the  reporting  staff  with  the  skills,  resources  and  support  required 
to  fulfill  the  surveillance  task,  as  well  as  to  ensure  the  maintenance  of  the  high 
performance  and  quality  of  the  data  generated  by  the  system. 

At  the  national  and  regional  team 

The  third  and  final  level  of  the  model  is  the  national  and  regional  disease 
surveillance  team  level.  A  national  team  is  created  to  oversee  the  implementation  process 
of  a  disease  surveillance  system.  In  order  to  provide  closer  support  to  the  surveillance 
sites  in  a  timelier  manner,  the  national  team  often  relied,  when  available,  on  regional 
teams  for  this  purpose. 

For  this  level,  the  researcher  will  describe  the  characteristics  of  these  teams  and 
the  functions  performed  to  ensure  the  surveillance  staff  at  the  local  sites  generated  the 
quality  and  timely  data  required.  The  outline  of  the  national  and  regional  team  level  is 
shown  in  Table  5. 

Table  5:  Outline  of  the  national  and  regional  team  level  concepts  in  the  Peruvian 
electronic  disease  surveillance  system  implementation  model,  as  identified  by  the 
primary  data  analysis. 


Team  characteristics 

•  Knowledge  and  skill 

•  Availability 

•  .-Continuing  education 

Team  role 

*  Monitoring 

•  Feedback 

•  Supervision 
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Team  Characteristics 


Research  participants  contended  that  the  essential  activities  of  training, 
monitoring,  supervision  and  feedback  could  be  accomplished  only  if  the  national 
surveillance  team  was  skilled,  highly  committed  (Int  17,  Int  18,  Int  21,  Int  15,  Int  3,  Int  8) 
and  available.  Participants  emphasized  the  importance  of  creating  a  team  to  lead  the 
implementation  process.  This  group  of  people  had  to  be  interested,  actively  involved  in 
surveillance,  convinced  of  its  usefulness  (Int  21),  and  experienced  in  surveillance  itself  or 
experienced  working  in  the  organization  in  which  the  system  would  be  implemented  (Int 
13,  Int  7,  Int  21,  Int  22,  Int  23). 

Knowledge  and  Skill 

Research  participants  noted  that  successful  disease  surveillance  implementation 
teams  maintained  a  high  level  of  technical  skills  and  high  morale  (Int  23),  which  was 
appreciated  by  the  surveillance  staff  at  different  reporting  sites  (Int  17,  Int  18,  Int  21,  Int 
15,  Int  3).  When  recruiting  personnel,  implementers  selected  people  motivated  to  work  in 
the  area  of  surveillance  (Int  23,  Int  15).  Inside  this  team,  at  least  one  person  had  to  have  a 
clear  vision  of  where  the  system  should  go  and  would  be  able  to  effectively  interact 
across  various  levels  in  hierarchical  organizations  (Inti).  The  national  team  would  be  the 
link  between  all  of  the  surveillance  sites  and  the  highest  level  authority  in  the 
organization  or  institution  (Int  1). 

The  skills  and  experience  of  the  national  surveillance  team  influenced  the  process 
of  implementation  (Int  1,  Int  9).  The  same  surveillance  system,  for  example,  was 
implemented  in  the  Peruvian  Navy,  Army  and  Air  Force  at  different  times  but  in  the 


order  indicated.  Research  participants  observed  that  the  implementation  process  was 
faster  and  easier  once  the  team  had  gained  more  experience  and  had  structured  the 
training.  They  mentioned  as  an  example  of  how  the  implementation  and  expansion  of  the 
system  in  the  Peruvian  Air  Force  was  completed  in  less  than  six  months  compared  to 
much  longer  times  for  implementation  and  expansion  in  the  other  two  branches.  The  Air 
Force  had  fewer  units,  but  the  Army,  which  had  many  more  units  than  the  Navy, 
consolidated  their  system  in  one  year  compared  to  the  three  years  it  required  for  the  Navy 
to  achieve  consolidation.  Participants  believed  that  the  experience  of  the  national 
implementing  team  with  the  surveillance  system  and  the  implementation  process 
contributed  to  a  more  efficient  and  effective  implementation  within  each  subsequent 
Force  (Int  1).  One  participant  shared  the  following  view  about  the  need  for  implementers 
to  have  epidemiological  knowledge  when  working  on  initiating  a  disease  surveillance 
system: 

El  equipo  que  va  a  implementor  debe  estar  comprometido  y  capacitado,  porque 
eso  es  lo  que  nos  hemos  podido  dor  cuenta;  quizas  el  sistema  funciono  porque 
nosotros  veniamos  de  la  parte  operativa  de  hacer  epidemiologia  y  cuando 
nosotros  empezamos  a  implementor,  podi'amos  responder  preguntas.  Que  pasaba 
si  nosotros  no  sabiamos  nada  de  epidemiologia  y  llegaba  acd  y  me  decian  sabes 
que,  implementa  un  sistema,  estaba  en  la  rnera  calle,  no  sabia.  (Int  11) 

[English  Translation :]  The  team  that  is  going  to  implement  [the  system  J  has  to  he 
committed  and  skilled,  because  what  we  have  realized,  maybe  is  that  the  system 
worked  because  we  came  from  the  operational  side  of  epidemiology,  and  when  we 
implemented  the  system,  we  could  respond  to  the  questions.  What  would  have 
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happened  if  we  did  not  know  about  epidemiology  and  we  came  here  and  were 
assigned  to  implement  a  system,  where  we  did  not  know  how  to  do  it.  (Int  II) 
Surveillance  staff  valued  the  high-level  surveillance  skills  and  knowledgeable  use 
of  the  surveillance  instrument  that  characterized  the  national  surveillance  team  (Int  1 ).  If 
an  electronic  tool  was  used,  technological  expertise  was  needed  to  maintain  the  tool  in 
terms  of  adding  or  changing  functions.  In  addition,  a  disease  surveillance  system 
required  effective  user  support  and  training.  Engineers  assigned  to  the  national 
surveillance  team  had  expertise  in  the  specific  software  program  used  to  develop  the  tool. 
If  the  tool  had  been  developed  by  another  organization,  the  engineers  would  have  needed 
access  to  the  programming  code  used  to  build  the  tool  (Int  12,  Int  8,  Int  15). 

It  was  important  that  at  the  national  level  the  disease  surveillance  team  had  an 
annual  work  plan  for  the  activities  they  needed  or  intended  to  perform  during  the  course 
of  the  year  (Int  4).  Research  participants  therefore  recommended  having  a  national  team 
that  was  exclusively  dedicated  to  surveillance  (Int  21,  Int  15,  Int  23).  The  reality, 
however,  was  that  the  team  had  other  assigned  tasks  unrelated  to  surveillance,  leaving 
team  members  little  time  to  develop  new  features  or  processes  that  would  improve  the 
surveillance  system  (Int  8).  Research  participants  shared  that  motivation  was  high  among 
the  team.  For  instance,  people  would  work  extra  hours  because  they  knew  the  job  was 
important,  particularly  when  a  need  arose  to  respond  to  disease  outbreaks  (Int  23). 
Participants  also  commented  that  when  there  was  political  will,  personnel  were  assigned 
to  the  surveillance  system  but  the  assignment  was  temporary  and  based  on  the  priorities 
of  the  person  in  the  highest  administrative  office  (Int  21). 

Availability 
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Research  participants  commented  that  the  perception  of  teams  available  to 
surveillance  staff  was  quite  important  (Int  15,  Int  18).  Disease  surveillance  systems  that 
functioned  effectively  were  characterized  by  constant  communication  and  the  presence  of 
a  surveillance  team  that  surveillance  staff  knew  that  they  could  call  when  problems  arose 
(Int  12,  Int  15,  Int  18).  Team  availability  and  responsiveness  to  the  concerns  of  staff  at 
any  level  contributed  to  the  development  of  trusting  relationships  (Int  7)  that  ultimately 
facilitated  a  more  effective  surveillance  system. 

Continuing  Education 

Research  participants  stated  that  if  the  national  implementation  team  was 
committed  to  the  surveillance  implementation  project  and  had  previous  experience  and 
training  in  disease  surveillance,  they  were  able  to  better  support  surveillance  staff  and 
respond  effectively  to  any  situation  that  might  have  arisen  (Int  1 1,  Int  17).  Therefore,  as 
the  implementation  process  evolved,  it  was  important  to  maintain  an  advanced  level  of 
training  for  surveillance  supervisors  as  well.  Supervisors  complained  about  not  having 
the  analytical  skills  to  analyze  the  data  they  had  been  monitoring  for  years  (Int  6). 
However,  addressing  this  issue  was  especially  challenging.  The  cost  of  continuing 
surveillance  education,  the  long  hours  of  work  experienced  by  supervisors,  and  the  small 
salary  of  supervisors  as  government  employees  (Int  15)  served  as  barriers  to  meeting  the 
continuing  education  needs  of  these  key  individuals. 

Team  Role 

The  team’s  functions  include  monitoring,  feedback,  supervision  and  support. 
Monitoring 
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While  training  was  key  to  providing  the  skills  required  and  motivate  the 
surveillance  staff,  system  monitoring  was  a  core  activity  in  the  maintenance  of  an 
effective  electronic  disease  surveillance  system  (Int  10,  Int  21,  lnt  9).  The  words  of  one 
research  participant  were  echoed  by  several  who  contended  that  system  success  depended 
on  close  monitoring  of  the  system  (Int  1 ,  Int  7,  Int  8): 

Tercer  pilar,  creo  que  es  la  monitorizacion  y  el  seguimiento,  an  sistema,  la 
implementacion  de  un  sistema  de  vigilancia,  creo,  no  puede  ser  exitoso  si  es  que 
no  existe  un  monitoreo  adecuado  del  sistema.  (Int.  1) 

[English  Translation:]  Third  pillar,  1  think  is  the  monitoring  and  follow-up,  of  a 
system;  the  implementation  of  a  disease  surveillance  system  cannot  be  successful 
if  there  is  no  adequate  monitoring  of  the  system.  (Int  1) 

Research  participants  advised  having  at  least  one  trained  person  assigned  to 
monitor  the  system,  including  assessing  data  quality  and  ensuring  reporting  accuracy  and 
timeliness  (Int  1,  Int  15.  Int  4,  Int  6,  Int  8).  Participants  specifically  mentioned  that  a 
well-functioning  system  depended,  in  part,  on  monitoring  all  aspects  of  reporting  disease 
surveillance  data  (Int  13).  The  designated  system  monitor  was  responsible  for 
communicating  with  surveillance  sites,  clarifying  issues  and  addressing  questions  from 
Staff  at  the  site,  and  engaging  in  supervisory  visits  (Int  8).  On  the  other  hand,  one  of  the 
factors  consistently  observed  in  a  failed  surveillance  system  was  a  lack  of  adequate 
system  monitoring  which  was  sometimes  related  to  assigning  monitoring  responsibility  to 
an  untrained  or  inadequately  trained  staff  member  (Int  1 ). 

National  surveillance  team  members  held  the  primary  responsibility  for  system 
monitoring.  Continuous  monitoring  was  difficult  to  maintain  when  these  personnel 
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became  overloaded  with  other  task  assignments  (Int  2,  Int  10,  Int  8).  Research 
participants  observed  that  when  no  one  monitored  the  system  and  contacted  surveillance 
staff  when  data  reporting  was  overdue  or  missing,  the  proportion  of  units  reporting 
disease  surveillance  data  by  a  given  deadline  date  dropped  from  80%  to  50%  or  60%  of 
the  units  (Int  1 1). 

Monitoring  was  important  to  identify  surveillance  areas  or  functions  that  were  not 
working  effectively.  This  was  accomplished  through  the  measurement  of  indicators  (Int 
22).  When  indicator  data  were  not  forthcoming,  training  (Int  15)  and  supervision  visits 
were  conducted  in  an  effort  to  respond  to  what  had  been  identified  as  a  problem  (Int  10, 
Int  21).  The  evaluation  of  the  system  was  important  in  an  effort  to  ultimately  assess  the 
effect  of  the  health  promotion,  and  disease  prevention  measures  that  had  been 
implemented  (Int  6,  Int  18), 

Feedback 

Feedback  was  one  more  function  of  the  national  surveillance  team  that  was 

deemed  essential  for  this  process.  Research  participants  agreed  that  feedback  was 

essential  in  both  directions.  From  the  end  users,  it  was  important  to  collect  suggestions  or 

requests  to  improve  the  tool  and  the  surveillance  processes  (Int.  1,  Int  10,  Int  12).  The 

national  surveillance  team  had  channels  established  that  the  end  users  utilized  to 

communicate  the  problems  they  had  with  the  system  or  the  tool  or  any  suggestions  for 

change  (Int  12).  On  the  other  end,  surveillance  staff  received  feedback  on  the  information 

they  collected  and  sent.  Study  participants  agreed  that  the  creation  of  bulletins  and  reports 

was  very  useful  (Int  9,  Int  1 3,  Int  1 8,  Int  2 1).  Reports  were  sent  to  every  region  in  the 

country  (Int  15),  weekly  (Int  4,  Int  15,  Int  18)  monthly,  and  annually  (Int  3).  These 

reports  were  sent  both  by  email  and  by  written  format  (Int  18).  Even  when  it  took  a  long 
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time  for  the  written  reports  to  reach  the  recipients,  surveillance  staff  at  the  surveillance 
sites  valued  having  a  print  document  to  show  their  supervisors  and  coworkers  the  work 
they  were  doing  (Int  18,  Int  3,  Int  4).  The  surveillance  staff  deemed  it  essential  to  see  the 
information  collected  in  a  way  that  was  useful  to  them;  research  study  participants 
believed  that  data  usefulness  was  helpful  in  motivating  surveillance  site  personnel  (Int  2, 
Int  3,  Int  14).  The  report  had  great  credibility  as  it  came  directly  from  the  organizations 
(Int  14). 

Research  participants  also  recommended  providing  end  users  feedback  on  their 
performance.  For  instance,  one  organization  created  a  ranking  of  the  sites  according  to 
their  overall  performance  and  timely  reporting.  During  the  national  “Epidemiology  Day’ 
they  congratulated  the  sites  that  performed  most  effectively  (Int  21,  Int  3).  An  additional 
strategy  they  used  was  to  have  monthly  meetings  to  share  information  and  congratulate 
staff  for  their  accomplishments  (Int  23). 

La  evidencia  de  que  el  sistema  permite  identificar  problemas,  la  evidencia  de  que 
la  informacidn  siendo  usada  adecuadamente,  genera  una  serie  de  acciones  que 
permiten  ademds  construir  o  generar  informacidn  util;  los  boletines  con  los 
reportes  de  la  situacion  por  cada  unidad  militar  ayudo  mucho  v  lo  otro  era ,  sacar 
un  ranking  de  cumplimiento  y  oportunidad  en  la  notificat  ion  para  que  el  personal 
se  sienta  motivado.  (Int  21) 

[English  Translation:]  The  evidence  that  the  system  allows  the  identification  of 
problems,  the  evidence  that  information  is  adequately  being  used,  generates  a 
series  of  actions  that  also  allows  generation  of  useful  information.  The  bulletins 
with  the  in  formation  of  the  health  situation  of  every  military  unit  helped  very 
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much,  the  other  thing,  was  to  have  a  ranking  with  the  reporting  performance 
which  made  the  personnel  feel  motivated,  (bit  21) 

Supervision 

Research  participants  stated  that  supervision  visits  should  accompany  monitoring 
and  evaluation  activities  whenever  possible  (Int  10).  Supervision  visits  depended  on  the 
budget  and  the  resources  available.  External  funding  supported  these  activities  in  military 
organizations  (Int  1 1,  Int  15,  lnt  9).  The  team  planned  to  supervise  units  based  on  the 
needs  of  the  system  (Int  10),  while  also  considering  their  geographic  accessibility  to  the 
units  (Int  15). 

Implementers  realized  that  when  they  supervised  a  unit,  the  reporting  increased 
and  the  act  of  supervision  motivated  the  surveillance  staff  in  several  ways  (Int  10).  The 
simple  fact  of  visiting  the  surveillance  staff  in  their  sites  helped  to  build  a  relationship 
with  them  (Int  9).  Listening  and  learning  about  their  local  problems,  their  difficulties  and 
challenges  were  also  considered  an  important  part  of  these  visits  (Int  10).  The  supervisory 
team  provided  information  of  what  was  expected  from  the  sites  before  supervising  them 
(Int  4,  Int  1 1 ).  There  was  a  tool  they  developed  for  the  supervision  visits  (Int  15).  During 
the  visit,  the  team  compared  the  data  reported,  which  appeared  on  the  electronic  system, 
with  the  one  registered  in  the  site  records  book  (Int  1 1,  Int  15,  Int  2).  If  any  problems 
were  identified,  the  team  retrained  the  surveillance  staff  to  clarify  any  misunderstandings 
(Int  2,  Int  9,  Int  11,  Int  15.  Int  18).  In  addition,  the  team  would  meet  with  all  the 
personnel  at  the  site  to  let  them  know  about  the  system  and  request  their  commitment  to 
the  task  (Int  9). 

This  visit  was  also  an  opportunity  for  the  team  to  learn  about  the  site,  the 
resources  in  place,  the  process  in  place,  and  the  surveillance  staff  s  morale.  This 
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understanding  was  important  to  the  team  to  determine  what  was  feasible  to  ask  from  the 
surveillance  staff  and  not  to  put  in  place  unfair  accountability  measures  (Int  11,  Int  15). 
After  the  supervision,  a  report  was  prepared  with  the  recommendations  if  they  found  any 
places  for  improvement  (Int  15).  The  team  was  careful  to  emphasize  the  objective  of  the 
supervision  as  a  way  to  improve  on  their  tasks  and  the  final  report  not  to  be  seen  as 
punishment  to  them  (  Int  5). 

Support 

Research  participants  believed  that  when  the  surveillance  regional  and  national 
level  maintained  constant  communication  with  the  surveillance  staff  at  the  site  level,  a 
high  motivation  environment  was  kept  at  the  site  (Int  10,  Int  13).  They  were  motivated 
when  they  received  support  to  fulfill  their  surveillance  tasks  from  the  national  and 
regional  surveillance  team  (Int  3).  The  supervision  visits  also  motivated  the  personnel  by 
facilitating  the  contact,  the  sharing,  and  the  building  of  a  relationship  between  the 
national  and  regional  surveillance  team  and  the  surveillance  staff  at  the  sites  (Int  9). 

In  summary,  the  national  and  regional  teams  should  be  composed  of  highly 
motivated  and  skilled  personnel.  The  critical  functions  of  training,  monitoring, 
supervision  and  feedback  address  the  critical  points  of  providing  the  skills  and  increasing 
the  motivation  of  the  reporting  staff  to  conduct  the  surveillance  task,  ensuring  high 
performance  at  the  surveillance  sites.  At  the  same  time,  the  national  team  has  the  critical 
function  of  informing  the  institutional  administrative  level  of  the  diseases  and  events 
under  surveillance.  By  presenting  the  analysis  of  the  data  collected  in  a  useful  way  to  the 
high  authorities  of  the  organization  or  institution,  they  can  translate  this  evidence  into 
measures  that  would  benefit  and  enhance  the  population  under  surveillance.  In  this  final 
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section,  the  researcher  will  explain  the  criticality  of  this  last  piece  to  the  sustainability  of 
the  system. 

Institutional  administrative  level 

Finally,  this  model  comes  back  to  the  first  level  described  in  order  to  conclude  the 
description  of  the  model.  The  institutional  administrative  level  is  essential  to  act  on  the 
information  that  has  been  collected  by  the  reporting  staff;  and  analyzed  and  formatted  by 
the  national  surveillance  team  to  inform  this  level. 

In  this  last  segment,  the  research  will  describe  the  final  critical  concept  that  has  to 
be  present  at  the  institutional  administrative  level  to  maintain  a  high  performing  system. 

Value  and  Action 

In  an  effort  to  implement  a  sustainable  disease  surveillance  system,  research 
participants  indicated  that  high-level  authorities  needed  to  be  able  to  value  and  use  the 
information  generated  by  the  system  by  ultimately  transforming  findings  into  decisive 
actions  that  addressed  disease  trends,  outbreaks,  or  other  situations  for  which  surveillance 
data  were  available.  It  was  their  actions  in  response  to  the  information  they  had  been 
provided  that  indicated  to  surveillance  staff  that  the  tasks  they  performed  were  valued 
and  worthy  of  continuation. 

Stakeholders  at  different  levels  of  the  surveillance  system  perceived  that 
demonstrating  the  usefulness  of  an  electronic  disease  surveillance  system  contributed  to 
system  success  in  several  ways.  The  system  provided  the  surveillance  team  with  valued 
disease  monitoring  information,  particularly  in  remote  areas  where  the  need  for 
surveillance  data  was  particularly  pronounced.  Disease  surveillance  enabled  them  to  not 
only  detect  and  respond  quickly  to  outbreaks,  but  also  to  more  effectively  plan,  budget 
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for,  and  manage  on-going  surveillance  activities  as  well  as  outbreaks  or  other  problems. 

It  also  enabled  them  to  tailor  health  promotion  and  disease  prevention  interventions  to 
populations  and  geographical  areas  in  need,  thus  potentially  affording  improved  disease 
morbidity  and  mortality. 

Disease  Detection  and  Rapid  Response 

Research  participants  appreciated  having  access  to  disease  surveillance  evidence 
in  different  geographical  areas  of  Peru.  Disease  surveillance  information  that  became 
available  for  the  first  time  in  some  remote  areas  of  the  country  (Int  3,  Int  15,  Int  17,  Int 
19)  was  particularly  valued,  given  past  experiences  with  disease  outbreaks  in  such  areas 
(Int  19). 

The  surveillance  system  enabled  staff  to  document  the  occurrence  of  health  events 
(  Int  21),  especially  outbreaks,  which  permitted  a  prompt  response  to  the  event  that  had 
previously  not  been  possible  (Int  1,  Int  6,  Int  13,  Int  16,  Int  19,  Int  10,  Int  23).  Armed 
Forces  research  participants  provided  an  example  of  a  foodbome  disease  outbreak  they 
were  able  to  detect  and  respond  to  in  a  population  of  new  recruits  who  were  initiating 
training  (Int  16-ST-PN).  Changes  in  kitchen  and  water  management  were  made  based  on 
food-borne  illness  outbreak  recommendations,  (Int  16)  as  well  as  changes  in  food 
preparation  and  serving  in  their  celebration  events  (Int  18).  A  research  participant 
offered  a  response  to  the  experience  in  the  following  comment. 

De  repente  esa  es  la  satisfaccidn,  de  poder  parar  de  repente,  la  enfermedad,  de 
poderla  detener  o  de  repente  de  tomar  las  medidas  necesarias  en  ese  momento ,  es 
el  bienestar  del  paciente.  La  verdad  que  a  ml  me  dio  mucha  satisfaccidn  de 
repente,  el  hecho  de  tener  mis  cadetes  alu  y  sentarme  a  curarlos,  que  se  yo.  Esa 
es  la  satisfaccidn,  de  repente  no  esperar  que  siga  avanzando.  Esa  es  la 
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satisfaction  propia  como  personal  de  salad,  al  margen  de  repente  que  puede 
haber  una  felicitation,  que  si  la  ha  habido,  reconozco.  Si.  he  tenido  felicitation 
de  parte  de  COSALE;  pero  al  margen  de  todo  eso,  es  la  satisfaction  personal  que 
uno  puede  tener.  (hit  17) 

[English  Translation:]  Maybe  that  is  the  incentive,  to  be  able  to  stop  the  disease, 
maybe  prevent  or  take  the  adequate  measures  in  that  moment  (to  stop  the 
progression  of  the  disease),  to  be  able  to  cure  my  soldiers.  That  is  my  incentive. 

To  prevent  the  progression  of  these  cases,  this  is  my  incentive  as  a  health  care 
worker,  besides  the  recognition  from  the  Army  Public  Health  department  that  1 
have  received,  (lnt  17) 

Planning,  Organizing,  and  Prevention 

In  addition  to  the  improved  detection  of  and  heightened  responsiveness  to  disease 
events,  research  participants  appreciated  the  planning  and  organization  that  a  disease 
surveillance  system  afforded.  They  highlighted  the  importance  of  having  a  system  that 
produced  data  that  decision-makers  actually  valued  and  used  (lnt  21,  lnt  23,  lnt  9,  lnt  20). 
Access  to  surveillance  information  enabled  decision-makers  to  better  prepare  themselves 
by  developing  appropriate  budgets  and  anticipating  the  need  for  and  requesting 
medication  (lnt  18,  lnt  14,  lnt  16);  programming  for  the  mobilization  of  human  resources 
(lnt  3);  and  planning  and  providing  disease  prevention  activities,  particularly  prior  to  the 
deployment  of  military  personnel  (lnt  21 ).  Finally,  surveillance  information  could 
document  the  impact  of  their  disease  prevention  and  health  promotion  activities,  some  of 
which  indicated  a  decreased  incidence  and  prevalence  of  certain  diseases  within  the 
populations  they  served  (lnt  19,  lnt  23,  lnt  22). 
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Figure  9:  Surveillance  site  level,  and  national  and  regional  surveillance  teams’  level. 


The  observation  that  the  organization  or  institution  responded  to  disease  events 
that  were  reported  served  as  an  incentive  for  surveillance  staff  to  continue  with  their  work 
(Int  16).  While  many  strategies  helped  the  reporting  staff  at  the  sites  to  increase  their 
commitment  to  the  surveillance  system,  evidence  of  data  use  by  system  personnel 
underscored  the  importance  of  the  system  and  showed  the  surveillance  staff  that  despite 
the  many  challenges  they  faced,  the  disease  surveillance  system  was  worth  maintaining. 
The  concepts  and  relationships  at  the  surveillance  site  level  and  the  national  and  regional 
team  level  are  summarized  in  Figure  9. 
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Outcome:  Timely  and  quality  information 
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Figure  10:  Peruvian  Electronic  Disease  Surveillance  System  implementation  model. 


In  summary,  value  has  to  be  perceived  at  the  institutional  administrative  level  in 
order  to  put  in  place  the  measures  to  respond  to  health  emergencies  and  to  plan,  organize 
and  prevent  the  occurrence  of  diseases  or  events  under  surveillance.  The  public  health 
actions  implemented  and  their  consequent  impact  on  the  population's  health  were 
considered  the  ultimate  motivator  for  the  health  care  workers  performing  the  surveillance 
tasks  at  the  surveillance  site  and  at  the  national  and  regional  team.  The  Grounded  theory 
informed  model  developed  by  the  researcher  using  the  qualitative  data  collected  in  Peru 
is  summarized  in  Figure  10  with  the  three  levels  identified. 
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CHAPTER  6:  TRIANGULATION  AND  REVISED  MODEL 


After  developing  an  electronic  disease  surveillance  systems  implementation 
model  (EDSSIM)  using  primary  data  collected  in  Peru,  the  researcher  incorporated 
triangulation  from  three  sources  to  validate  and  refine  the  model.  The  steps,  procedures 
and  products  of  each  data  source  explored  in  this  study  are  summarized  in  Figure  1 1 . 

Comparing  the  concepts  across  data  sources 

The  researcher  compared  the  results  of  the  electronic  disease  surveillance  systems 
implementation  model  to  those  obtained  using  three  methodologies:  (1 )  the  systematic 
review  of  the  literature  on  health-related  program  implementation  factors  in  developing 
countries,  (2)  the  gray  literature  search  for  implementation  factors  of  electronic  disease 
surveillance  systems  in  developing  countries,  and  (3)  the  experts  consensus  report  on 
implementation  factors  related  to  electronic  disease  surveillance  systems  in  developing 
countries  (Tables  6.  7  &  8). 

The  researcher  identified  three  organizational  levels  associated  with  the 
implementation  of  the  Peruvian  EDSSIM.  The  researcher  explored  every  level  for 
concordances  and  discordances  in  the  concepts  and  the  findings  of  the  different 
methodologies  incorporated  in  this  study. 
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Figure  11:  Steps,  Procedures  and  Products  of  the  study  used  to  create  a  refined  model. 
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At  the  administrative  level  of  the  model 

Comparison  to  the  systematic  literature  review 

When  comparing  the  results  of  the  EDSS1M  to  those  of  the  systematic  literature 
review  described  in  the  background  chapter,  the  researcher  noted  most  of  the  factors 
described  in  the  literature  review  were  present  or  were  part  of  concepts  in  the  electronic 
disease  surveillance  systems  implementation  model.  The  concepts  identified  by  the 
systematic  literature  review  that  were  present  in  the  model,  included  “engaging  key 
stakeholder  groups”  (awareness  of  the  significance),  “accountability  policies” 
(legislation),  “program  leadership”  and  “political  environment”  (commitment  to 
ownership),  “value  and  action”  (capacity  building),  and  “donor  and  external  support" 
(financial  commitment). 

The  systematic  literature  review  presented  the  concept  of  “engaging  key 
stakeholders  groups”  as  the  Peruvian  EDSSIM  model  concept  of  “awareness  of  the 
significance"  at  the  highest  organizational  level  of  the  system.  System  implemented 
would  engage  the  key  stakeholder  group  as  a  consequence  of  educating  high-level 
administrative  personnel  in  the  high  administrative  level  in  an  organization  about  the 
importance  of  surveillance.  The  “accountability  policies”  subcategory  included  the 
government  legal  and  regulatory  aspects  for  the  program  that  in  the  Peruvian  EDSSIM 
was  discussed  as  the  “Legislative  action”  concept.  The  “program  leadership”  and 
“political  environment”  subcategories  underscored  the  importance  of  active  leadership  at 
the  government  level  and  the  value  of  committed  national  political  leaders  in  support  of  a 
program.  This  subcategory  contributed  to  build  the  “organizational  commitment”  concept 
in  the  Peruvian  EDSSIM. 
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The  “donor  and  external  support”  subcategory  in  the  literature  referred 
specifically  to  the  “financial  support”  concept  in  the  mpdel.  The  term  “donor  support”  to 
refer  to  the  “financial  support"  concept  may  have  been  used  by  authors  in  the  literature 
because  most  of  the  program  implementation  experiences  in  developing  countries 
published  in  the  literature  related  to  systems  whose  funding  was  supported  by  external 
organizations.  The  “capacity  building”  subcategory  was  included  in  a  broader  Peruvian 
EDSSLM  concept  named  “value  and  action.”  According  to  the  Peruvian  EDSSIM,  if  an 
organization  recognized  the  value  of  the  surveillance  information  collected,  personnel  at 
high  administrative  levels  would  use  this  information  to  build  the  capacity  needed  to 
support  the  system. 

The  systematic  literature  review  did  not  contribute  to  any  additional  concepts  at 
this  level.  Conversely,  several  concepts  at  the  high  administrative  level  of  the  model  not 
identified  in  the  systematic  literature  review  were  present  in  the  Peruvian  EDSSLM. 

These  concepts  included  “organizational  need,”  “technical  assistance.”  and 
“coordination.”  Since,  of  necessity,  the  focus  of  the  systematic  literature  review  was 
health-related  programs  concepts  such  as  “organizational  need"  might  fail  to  be  identified 
due  to  the  differing  focus  and  needs  associated  with  these  programs.  Ln  terms  of 
“technical  assistance”,  the  capacity  of  each  organization  implementing  a  program  might 
vary  as  would  its  consequent  need  for  such  assistance.  The  “coordination"  required  to 
implement  a  program  would  depend  on  the  organizational  levels  and  sectors  involved. 
Electronic  diseases  surveillance  programs  were  typically  national  programs  for  which 
coordination  was  essential. 

Comparison  to  the  gray  literature  review 
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At  the  administrative  level.  Findings  form  the  gray  literature  coincided  with  the  electronic 
disease  surveillance  systems  implementation  model  in  several  concepts.  Although  the 
names  of  some  concepts  differed  between  the  model  and  the  gray  literature,  they  shared 
similar  ideas.  Commonalities  between  the  concepts  in  both  methodologies  included 
“advocacy"  (awareness  of  the  significance),  “political  commitment”  (commitment  to 
ownership),  legislation,  financial  commitment,  “development  of  partnership  and 
networks,"  and  “communication”  (coordination),  “development  of  partnership  and 
networks”  (technical  assistance),  and  “surveillance  strategy”  (value  and  action).  The 
statements  in  quotations  and  parentheses  relate  to  the  gray  literature  and  the  Peruvian 
EDSSIM  respectively. 

“Advocacy”  was  presented  in  the  gray  literature  as  an  important  tool  for  creating 
awareness  of  the  importance  of  surveillance  among  stakeholders.  Political  commitment 
was  mentioned  in  the  literature  as  a  component  of  surveillance  implementation  that 
reflected  the  support  of  the  high  level  administrators.  It  was  described  in  the  Peruvian 
EDSSIM  as  “commitment  to  ownership."  Legislation  and  financial  commitment  were 
concepts  described  in  similar  ways  by  both  methodologies,  but  more  extensively  by  the 
electronic  disease  surveillance  systems  implementation  model.  In  the  gray  literature,  the 
concept  of  “coordination”  was  described  by  two  concepts:  “development  of  partnerships 
and  networks”  and  “communication.”  “Development  of  partnerships  and  networks”  with 
public  and  private  agencies  for  resource  mobilization,  formulation  of  training  policy,  and 
development  of  standard  and  guidelines  required  strong  coordination  mechanisms. 
“Communication”  among  the  different  stakeholder  groups  involved  all  levels  of  the 
health  system.  The  “development  of  partnerships  and  networks”  in  the  gray  literature 
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review  also  addressed  part  of  the  “technical  assistance”  concept,  by  describing  the 
support  derived  from  partnerships  with  international  organizations.  A  component  of  the 
concept  in  the  model  of  “value  and  action”  was  addressed  in  the  gray  literature  review  by 
the  “surveillance  strategy”  concept.  Consistent  with  the  model,  authors  of  the  gray 
literature  underscored  decision  makers’  use  of  disease  surveillance  information  and  the 
importance  of  identifying  the  utility  of  the  system  in  the  early  stage  of  the 
implementation  process. 

The  gray  literature  review  contributed  to  insights  germane  to  the  high 
administrative  level  by  adding  the  important  concept  of  having  a  “strategic  vision”  to 
ensure  an  organization's  priorities  were  addressed  when  a  system  was  implemented. 
Moreover,  the  gray  literature  results  noted  the  importance  of  moving  from  a  vertical 
systems  approach  toward  the  integration  of  surveillance  activities  into  the  organization. 
An  additional  concept  contributed  by  the  gray  literature  that  was  consistent  with  the 
Peruvian  EDSSIM  was  the  need  for  an  adequate  organizational  structure  with  clear  roles 
and  responsibilities  for  surveillance  within  the  system. 

Comparison  to  the  expert  consensus  workshop 

Findings  from  the  expert  consensus  workshop  concurred  at  the  administrative 
level  with  the  concepts  of  the  Peruvian  EDSSIM  either  by  fully  or  partially  describing 
them.  The  concept  of  “organizational  need”  was  partially  described  by  workshop 
participants  as  the  determination  of  an  organization’s  motivation  and  direction.  The 
“usefulness  at  all  levels”  concept  covered  in  part  the  concept  of  “value  and  action"  in  the 
model,  indicating  that  recognition  of  the  importance  of  information  generated  by  the 
system  would  be  the  first  step  to  action  that  was  based  on  that  information.  The  experts 
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described  “commitment  to  ownership”  as  strong  executive  support  and  “political  will." 
Legislation,  technical  assistance,  and  financial  commitment  were  concepts  fully  agreed 
upon  by  both  methodologies.  Conversely,  the  “coordination"  concept  elaborated  on  in  the 
model  was  only  partially  addressed  in  the  workshop  report  by  acknowledging  the  need  to 
secure  and  share  the  surveillance  data. 

The  expert  consensus  workshop  contributed  important  concepts  to  the  highest 
administrative  level  of  the  Peruvian  EDSS1M.  For  instance,  the  value  and  need  for  a 
“strategic  vision  and  plan”  coincided  with  the  gray  literature  review  results.  The  experts 
agreed  on  the  importance  of  developing  a  strategic  plan  that  included  monitoring  and 
evaluation,  workforce  development,  and  steps  to  sustainability.  This  plan,  according  to 
the  consensus  workshop,  should  be  developed  after  the  implementers  had  performed  an 
assessment  of  country  readiness  for  an  electronic  disease  surveillance  system,  including 
the  infrastructure  and  capacity  of  the  organization  to  support  such  a  system.  Another 
element  the  experts  recommended  to  be  considered  at  this  level  is  to  start  with  a  pilot  or 
demonstration  project  to  ensure  acceptability  of  the  system  before  expanding  the  system. 
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Table  6:  Comparison  of  triangulated  concepts  at  the  ADMINISTRATIVE  LEVEL. 
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At  the  surveillance  staff  level 

Comparison  to  the  systematic  literature  review 

At  the  surveillance  staff  level,  the  systematic  literature  review  identified  the 
elements  of  “training,”  “program  flexibility,”  “human  resources,”  “engaging  key 
stakeholder  groups,”  “social  environment,”  and  “infrastructure  and  capacity."  “Training” 
characteristics  such  as  the  format  of  a  training  session,  and  the  format  of  training 
materials  were  described  in  the  literature  review.  The  researcher  explored  this  concept 
more  extensively  while  developing  the  Peruvian  EDSSIM.  In  the  model,  “training”  was 
emphasized  as  a  continuous  activity  that  should  be  available  to  and  conducted  at  all  levels 
of  the  organization.  “Flexibility”  was  described  in  the  systematic  literature  review  as  an 
attribute  of  the  tool  used  for  data  collection  as  well  as  the  processes  in  place  in  order  to 
increase  acceptability  of  the  system.  Consistent  with  the  model,  “human  resources”  was 
noted  in  the  literature  review  as  the  most  common  challenge  for  health  program 
implementation.  The  literature  review  pointed  to  “infrastructure  and  local  capacity"  as 
important  factors  for  program  implementation,  which  were  recognized  in  the  model  as 
the  existence  of  appropriate  resources  to  conduct  surveillance  duties.  A  favorable  “social 
environment”  for  implementation  was  described  as  the  presence  of  trust  between  the  staff 
and  a  caring  ethos  that  extended  to  the  community.  This  concept  intersected  with  the 
concepts  in  the  Peruvian  EDSSIM  of  “awareness  of  the  importance  of  surveillance,” 
“creation  of  an  on-site  surveillance  team.”  and  “support  by  the  national  and  regional 
teams.” 

The  systematic  literature  review  did  not  posit  the  concepts  of  “simplicity”  and 
“standardization”  that  were  included  in  the  model,  which  might  be  due  to  the  fact  that  the 
review  on  health  related  programs  implementation  only  included  three  papers  that 
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reported  on  disease  surveillance  systems  experience.  One  of  the  papers  that  specifically 
addressed  surveillance  systems  in  the  systematic  literature  review  addressed  the  training 
on  data  analysis. 

Comparison  to  the  gray  literature  review 

There  was  strong  concurrence  between  the  gray  literature  at  the  surveillance  site 
level  and  those  identified  in  the  model.  While  the  model  provided  a  more  extensive 
explanation  and  analysis  of  concepts  at  this  level,  the  researcher  identified  similarities  in 
both  methodologies.  The  “human  resources"  concept  in  the  gray  literature  comprised  the 
“skilled  and  motivated  human  resources”  concept  in  the  electronic  disease  surveillance 
systems  implementation  model.  The  organization  of  the  system  (director’s  support), 
resources,  surveillance  tool,  (simplicity),  flexibility,  and  standards  and  guidelines  were 
important  concepts  at  the  surveillance  site  level.  The  concepts  of  “simplicity”  and 
“flexibility”  specifically  related  to  the  surveillance  tool  in  the  gray  literature  review, 
whereas  the  model  addressed  these  two  concepts  in  surveillance  processes  as  well. 

An  important  contribution  by  the  gray  literature  review  was  the  “data  analysis” 
concept.  Authors  of  publications  in  the  gray  literature  considered  “data  analysis”  a  key 
element  at  the  surveillance  site  level,  as  this  skill  would  empower  surveillance  staff  to 
understand  the  data  collected  and  appropriately  respond  to  the  information  generated.  The 
lack  of  this  concept  in  the  model  might  be  due  to  the  marked  emphasis  in  Peruvian 
surveillance  systems  studied  to  assure  that  reporting  occurred  in  all  sites.  Even  when 
there  was  recognition  that  data  analysis  skills  are  important  for  the  reporting  staff,  the 
main  objective  of  the  implementers  was  to  first  have  a  high  reporting  rate. 
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Comparison  to  the  expert  consensus  workshop 

There  was  strong  concordance  between  the  expert  consensus  workshop  and  the 
electronic  disease  surveillance  systems  implementation  model  at  the  surveillance  site 
level.  The  “skilled  and  motivated  human  resources  concept"  in  the  model  was  explained 
by  experts  at  the  consensus  workshop  by  the  concepts  of  “trained  and  available 
workforce,"  and  the  development  of  a  “performance  improvement  plan"  aimed  to 
motivate  surveillance  staff.  The  “resources”  concept  in  the  model  was  linked  to  “financial 
commitment"  factor  explained  during  the  workshop  as  a  commitment  that  would  assure 
that  the  resources  needed  were  in  place  for  the  implementation  of  the  program.  The 
system’s  “flexibility"  and  “simplicity"  concepts  in  the  model  were  noted  by  the  experts 
as  a  system  that  was  “flexible  and  adaptable,"  and  “intuitive  to  users." 

The  consensus  workshop  experts  did  not  mention  the  concept  of  “director’s 
support”  or  “standardization"  of  the  guidelines  and  processes.  However,  the  participants 
identified  data  analysis  training  as  one  of  the  main  gaps  in  the  developing  world. 
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Table  7:  Comparison  of  triangulated  concepts  at  the  SURVEILLANCE  SITE  LEVEL. 
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The  national  and  regional  team  organizational  level 
Comparison  to  the  systematic  literature  review 

At  the  national  and  regional  team  level  of  organization,  the  researcher  identified 
subcategories  that  emerged  from  the  systematic  literature  review  as  “training,”  “human 
resources,”  “engaging  stakeholder  groups,”  “infrastructure  and  local  capacity,”  “monitoring  and 
evaluation,”  “supervision,”  “communication,”  and  "incentives”.  The  “communication” 
subcategory  from  the  literature  review  mainly  described  “feedback  mechanisms”  used  to  inform 
staff  about  their  performance,  as  well  as  the  “coordination”  required  between  levels  of  the 
program.  The  researcher  identified  both  “feedback”  and  “coordination”  in  the  model.  “Engaging 
stakeholder  groups”  was  described  in  the  systematic  literature  review  as  the  closest  subcategory 
to  the  model  concepts  of  “motivated  staff’  and  “awareness  of  the  significance.”  “Engaging 
stakeholder  groups”  at  all  levels  of  the  systems  consisted  on  motivating  staff  and  educating 
stakeholders  on  the  importance  of  surveillance  in  order  to  achieve  commitment. 

One  concept  that  deserves  further  explanation  is  “accountability  policies”.  Both  the 
systematic  literature  and  grey  literature  reviews  indicated  that  a  lack  of  mechanisms  for 
accountability  could  jeopardize  program  implementation.  While  the  EDSSIM  reflected 
recognition  of  the  importance  of  having  legislation  in  place,  as  a  means  of  staff  accountability  for 
surveillance  reporting,  it  was  advised  that  implemented  act  with  caution.  Data  that  informed  the 
model  revealed  that  surveillance  staff  appreciated  steps  taken  to  notify  them  about  a  lack  of 
reporting  before  strict  measures  were  taken.  Surveillance  staff  members  were  limited  in  number 
and  had  many  other  competing  duties.  Therefore,  the  national  and  regional  teams  were  advised  to 
investigate  the  reason  for  reporting  problems.  According  to  these  data,  initiating  strict  measures 
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led  surveillance  staff  to  resent  the  national  team.  It  also  meant  that  they  sometimes  engaged  in 
reporting  only  due  to  fear  to  being  sanctioned. 

The  use  of  incentives  was  another  concept  that  was  not  consistent  among  the  three  major 
sources  of  data.  Both  the  systematic  literature  review  and  the  gray  literature  review,  for  instance, 
recommended  the  use  of  financial  and  non-financial  incentives  for  adherence  to  surveillance 
program  duties.  Conversely,  experts  involved  in  the  consensus  workshop  and  data  that  informed 
the  Peruvian  EDSSIM  agreed  on  using  only  non-financial  incentives  because  of  the  program 
sustainability  issues  in  a  resource  constrained  setting. 

The  systematic  literature  review  did  not  reveal  any  additional  concepts  at  the  national  and 
regional  team  level. 

Comparison  to  the  gray  literature  review 

At  the  national  and  regional  team  level,  there  was  strong  concurrence  between  the  gray 
literature  review  and  the  electronic  disease  surveillance  systems  implementation  model  on 
concepts  of  “human  resources  skilled  and  motivated”  with  the  gray  literature  review  emphasizing 
the  analytical  skills  of  this  group.  As  for  “resources,”  “training,”  “monitoring  and  evaluation,” 
“supervision,"  and  “feedback”  the  results  were  similar.  The  discordances  found  in  the  concepts 
of  “accountability  policies”  and  “use  of  incentives”  are  described  above. 

The  gray  literature  review  did  not  add  any  further  concepts  at  this  level. 

Comparison  to  the  expert  consensus  workshop 

At  this  level,  during  the  consensus  workshop  the  experts  discussed  a  few  concepts  that 
comprised  the  ones  described  by  the  electronic  disease  surveillance  systems  implementation 
model.  For  instance,  the  “training”  concept  in  the  workshop  applied  both  for  the  presence  of 
“trained  and  available  human  resources,”  and  to  the  specific  “training”  concept.  The 
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“performance  improvement  plan”  as  explained  during  the  consensus  workshop,  recognized  the 
“motivation”  component,  the  “feedback”  functions,  and  the  “use  of  incentives”  at  this  level.  The 
monitoring  activities  were  described  by  both  methodologies,  and  the  resources  concept  was 
linked  to  the  financial  commitment  concept  defined  by  experts. 

The  concepts  that  were  not  mentioned  during  the  expert  consensus  workshop  were  the 
accountability  and  the  supervision  concepts. 

The  experts’  consensus  workshop  did  not  add  any  additional  concepts  at  this  level. 
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Table  8:  Comparison  of  triangulated  concepts  at  the  NATIONAL  REGIONAL  TEAM  LEVEL. 
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Advantages  of  using  Grounded  theory  model  in  this  study 

The  grounded  theory  methodology  was  fundamental  to  understanding  the 
complexity  of  the  factors  related  to  the  implementation  of  disease  surveillance  systems 
and  to  fully  describe  the  relationships  of  these  concepts  at  every  level  of  the  model. 
Through  this  methodology  the  researcher  was  also  able  to  identify  concepts  not 
recognized  by  any  other  methodology  used  in  this  study. 

At  the  Administrative  level,  concepts  such  as  “organizational  need,”  “awareness 
of  the  significance,”  and  “value  and  action”  were  only  fully  characterized  using  grounded 
theory  methodology.  Some  of  the  characteristics  of  the  “organizational  need”  concept 
were  briefly  described  by  the  experts’  consensus  workshop.  This  included  the  importance 
of  determining  the  country’s  motivation  and  direction,  and  identifying  the  country 
surveillance  needs. 

Different  dimensions  in  the  concept  of  “awareness  of  the  significance”  were 
described  by  the  three  other  methodologies,  failing  to  fully  characterize  it.  The  systematic 
literature  review  identified  “engaging  key  stakeholders  groups”  at  the  high  administrative 
level,  while  the  grey  literature  review  highlighted  the  importance  of  using  “advocacy”  for 
the  surveillance  system  at  this  high  level  as  well. 

The  “value  and  action”  concept  that  links  two  of  the  levels  in  the  model  was 
understood  in  some  of  its  dimensions  by  the  three  other  methodologies.  The  systematic 
literature  review  reported  the  importance  of  the  action  component  by  recognizing 
capacity  building  as  one  of  its  subcategories.  One  of  the  documents  in  the  grey  literature 
review  noted  the  importance  of  acting  on  the  information  collected  as  part  of  their 
surveillance  strategy.  The  experts’  consensus  workshop  report  included  the  value 
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component  of  this  concept  by  acknowledging  the  importance  of  having  a  system  useful  at 
all  levels,  which  included  the  administrative  and  the  surveillance  site  level. 

At  the  surveillance  site  level,  the  grounded  theory  methodology  revealed  the 
director’s  support  concept  that  the  other  three  methodologies  failed  to  capture.  However, 
the  systematic  literature  review  discussed  “social  environment”  as  an  important 
consideration.  This  subcategory  described  the  importance  of  trust  among  the  staff  who 
worked  in  the  program.  The  gray  literature  review  pointed  to  the  organization  of  the 
system  as  a  key  element  related  to  the  director’s  support  concept.  This  review 
emphasized  creating  an  adequate  organizational  structure  with  clear  roles  and 
responsibilities,  as  well  as  developing  team  spirit  and  trust  among  stakeholders. 

At  the  national  and  regional  team  level,  the  electronic  disease  surveillance 
systems  implementation  model  made  an  important  clarification  regarding  the 
accountability  concept  that  other  methodologies  failed  to  identify. 

Concepts  added  to  the  model 

The  systematic  literature,  the  gray  literature  review  and  the  experts’  consensus 
workshop  were  essential  to  confirm  and  validate  the  concepts  identified  by  the  electronic 
disease  surveillance  systems  implementation  model.  However,  these  methodologies  also 
proved  useful  to  identify  concepts  that  did  not  arise  with  the  electronic  disease 
surveillance  systems  implementation  model. 

At  the  administrative  level,  the  gray  literature  review  and  the  experts’  consensus 
workshop  reported  on  the  concept  of  “strategic  vision  and  plan"  as  an  important  addition 
to  this  level.  Both  methodologies  agreed  on  the  criticality  of  having  a  strategic  vision  and 
plan  to  ensure  the  system  was  implemented  following  the  organization’s  interests  and 
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needs,  and  therefore  facilitate  country  ownership  of  the  system.  At  this  same  level,  the 
gray  literature  emphasized  the  importance  of  prioritizing  the  list  of  diseases  under 
surveillance.  Prioritization  of  the  list  of  events  under  surveillance  was  discussed  within 
the  simplicity  concept  at  the  electronic  disease  surveillance  systems  implementation 
model.  However,  this  finding  supported  the  decision  of  identifying  “prioritization  of 
events  under  surveillance"  as  a  distinct  concept.  Finally,  the  gray  literature  added  the 
concept  of  developing  an  adequate  organizational  structure  for  the  system  by  outlining 
clear  roles  and  responsibilities  for  every  level  of  the  model.  The  researcher  included  this 
concept  at  the  administrative  level  since  the  development  of  these  roles  occurs  at  this 
level. 

At  the  surveillance  site  level,  all  three  methodologies  concurred  on  the  inclusion 
of  the  data  analysis  concept.  While  the  three  methodologies  concurred  with  the  electronic 
disease  surveillance  systems  implementation  model  on  the  need  for  data  analysis  training 
at  this  level,  the  gray  literature  review  expanded  on  this  concept  by  highlighting  the 
importance  of  developing  automated  analysis  features  within  the  tool  and  the  provision  of 
clear  technical  guidance  to  facilitate  this  function  for  the  surveillance  staff.  Based  on  this 
finding,  the  researcher  decided  to  describe  the  data  analysis  as  a  concept  beyond  the 
training  aspect  of  it. 

At  the  national  and  regional  team  level,  no  new  concepts  were  identified  by  the 
other  three  methodologies. 

In  summary,  the  new  revised  model  included  two  more  concepts  at  the  high 
administrative  level  (Figure  12),  and  two  more  concepts  at  the  surveillance  site  level 
(Figure  13). 
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Figure  12:  Institutional  administrative  level  of  the  revised  model.  In  yellow  the  concepts 
added  after  triangulation  of  data  sources. 
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Figure  13:  Surveillance  site  level  and  National  and  regional  team  level  of  the  revised 
model.  In  yellow  the  concepts  added  after  triangulation  of  data  sources. 


Recommendations 

The  revised  electronic  disease  surveillance  systems  implementation  model 
presents  key  elements  to  consider  when  there  is  either  a  need  to  implement  an  electronic 
disease  surveillance  system  in  resource  limited  settings  or  a  desire  to  assess  a  system 
already  in  place.  This  study  advises: 

1 .  The  administrative  authorities  at  the  institution  or  organization  where  the  system 
will  be  implemented  must  be  committed  to  the  system.  To  achieve  this 
commitment,  epidemiology  leaders  or  in  some  cases  external  organizations  need  to 
advocate  for  the  importance  of  the  surveillance  system,  and  demonstrate  this  system 
is  truly  needed  by  the  institution  or  organization.  Education  on  the  importance  of 
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surveillance  is  often  required  to  develop  awareness  at  this  level  and  ensure 
commitment  to  the  full  implementation  process  and  ultimately  the  sustainability  of 
the  system. 

2.  Implemented  need  to  ensure  that  the  institutional  administrative  commitment 
achieved  for  the  disease  surveillance  system  translates  into  the  following  actions: 

a.  The  development  of  a  strategic  vision  and  plan  that  reflects  the  real  needs  of 
the  organization  and  maps  out  the  sustainability  strategy  for  the  disease 
surveillance  system.  This  plan  should  include  a  monitoring  and  evaluation 
framework  and  an  articulated  workforce  development  plan. 

b.  The  creation  or  update  of  legislation  to  support  the  implementation  and 
functioning  of  the  disease  surveillance  system  that  also  provides  a  means  of 
accountability  of  the  required  processes  and  tasks.  The  content  of  this 
document  must  include  the  organizational  structure  of  the  system  with  clear 
roles  and  responsibilities  for  each  level,  all  the  processes  required,  the 
monitoring  and  evaluation  activities  of  the  inputs,  processes  and  outcomes, 
and  the  use  of  the  information.  This  document  should  also  explicitly  describe 
the  coordination  mechanisms  within  the  different  levels  of  the  organization, 
between  organizations  that  perform  similar  or  complementary  surveillance 
activities  within  the  same  the  institution,  and  between  the  institution  and  the 
external  organizations  providing  technical  and/or  financial  support. 

c.  The  administrative  level  should  ensure  the  financial  support  required  for  the 
implementation  process  by  committing  the  institution  to  the  required  funding, 
by  identifying  an  external  donor,  by  establishing  collaboration  efforts  with 
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other  institutions,  or  by  using  a  strategy  that  combines  them.  The  amount  and 
duration  of  this  support  should  be  clearly  stated. 

d.  Adequate  technical  support  has  to  be  secured  for  the  implementation  of  the 
system,  either  by  assigning  skilled  staff  from  the  institution  or  through  an 
agreement  with  an  organization  that  provides  the  expertise  required. 

e.  The  administrative  level  should  guarantee  the  dissemination  of  this  document 
to  all  stakeholders’  groups  and  surveillance  sites.  The  implemented'  team 
may  use  this  document  as  an  important  tool  to  create  awareness  of  the 
importance  of  the  disease  surveillance  system  being  implemented  and  to 
demonstrate  the  commitment  of  the  highest  authority  in  the  institution  to  the 
system. 

3.  The  implementers  must  ensure  that  there  is  a  surveillance  network  composed  of 
national,  regional  and  if  feasible  district  surveillance  teams.  The  national  team  must 
be  a  skilled  and  motivated  group  with  strong  commitment  to  the  implementation  of 
the  system.  This  team  should  use  a  top  down  approach  to  train  and  educate  the 
lower  level  teams  and  instill  the  same  level  of  commitment.  These  lower  level 
teams  once  trained  will  be  directly  responsible  for  the  activities  required  to  prepare 
fully  functioning  surveillance  sites  within  their  jurisdiction.  The  national  team  is 
directly  responsible  for  ensuring  the  high  authority  level  stays  committed  and 
supports  this  process,  as  well  as  to  respond  to  the  needs  of  the  lower  level  regional 
and  district  teams.  The  regional  and  district  surveillance  teams  are  responsible  for 
directly  responding  to  the  needs  of  the  reporting  staff  at  the  surveillance  sites  or  by 
requesting  assistance  at  the  national  level. 
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4.  The  national  surveillance  team  must  ensure  that  the  following  functions  are 
performed  to  support  the  reporting  staff  at  the  surveillance  sites: 

a.  Following  the  monitoring  and  evaluation  framework,  the  national  or  regional 
team  must  make  sure  all  the  processes  to  generate  the  surveillance  information 
are  taking  place.  This  monitoring  activity  should  be  used  to  increase  the 
quality  of  the  data  by  identifying  the  retraining  needs  of  the  personnel,  and 
refinements  required  in  the  surveillance  processes  and  the  surveillance  tool. 

b.  The  information  generated  should  be  used  to  provide  feedback  of  their 
activities  to  the  surveillance  staff  and  the  high  administrative  level  and  as 
proof  of  the  usefulness  of  the  system,  which  contributes  to  the  continuance  of 
support. 

c.  Through  the  monitoring  and  evaluation  activities,  sites  should  be  prioritized 
for  supervision.  The  supervision  visits  represent  an  opportunity  for  retraining 
surveillance  staff;  learning  the  reality  of  the  site,  its  challenges  and 
opportunities;  and  developing  a  close  relationship  with  reporting  staff  to 
improve  future  communication. 

d.  The  national,  regional,  and  district  teams  should  represent  a  support  source  for 
the  reporting  staff  at  the  surveillance  sites.  Therefore,  sanctioning  for  lack  of 
timely  reporting  should  be  left  as  a  final  resource.  Learning  the  reasons  for 
low  performance  and  addressing  them  encourages  reporting  staff  to  confide 
the  obstacles  faced  and  surpass  them. 

5.  The  regional  and  district  surveillance  teams  in  coordination  with  the  national  teams 
should  prepare  on-site  functional  surveillance  teams  at  every  site  instead  of 
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assigning  this  task  to  a  few  people.  These  local  surveillance  teams  need  an  adequate 
environment  to  function,  the  skills  required  to  perform  the  surveillance  tasks,  the 
motivation  to  conduct  this  task,  and  a  feasible  task  considering  their  other  multiple 
tasks. 

6.  The  provision  of  an  adequate  environment  for  the  reporting  staff  include  providing 
the  resources  required  to  perform  the  task,  and  creating  awareness  of  the 
significance  of  the  disease  surveillance  system  in  all  health  care  personnel  including 
the  director  of  the  surveillance  site  to  support  this  task. 

a.  Maximizing  the  use  of  the  resources  already  in  place  is  critical,  either  by 
identifying  the  resources,  updating  them  or  securing  the  administrative 
permits  that  guarantee  the  reporting  staff  has  access  to  them.  Developing 
collaborations  with  other  surveillance  efforts  within  the  same  institution  or 
external  organizations  has  proved  to  be  an  effective  strategy  to  secure 
resources  and  to  increase  the  efficiency  of  its  use. 

b.  Educating  health  care  personnel  on  the  importance  of  the  task  conducted  by 
this  onsite  team  is  crucial  to  create  an  environment  of  support  at  the 
surveillance  site  level. 

7.  The  provision  of  the  adequate  skills  requires  a  continuous  and  comprehensive 
training  strategy  tailored  to  the  needs  of  the  reporting  staff.  Innovation  in  the 
delivery  strategies  is  especially  needed  to  be  able  to  reach  the  surveillance  staff 
even  in  the  most  remote  areas  and  give  a  better  use  to  the  resources  available.  Data 
analysis  skills  should  be  a  priority  once  the  reporting  staff  is  comfortable  with  using 
the  system. 
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8.  A  performance  improvement  plan  must  be  in  place  to  ensure  the  level  of  motivation 
of  the  staff  remains  high.  Finding  the  right  incentives  that  can  become  a  sustained 
practice  is  essential  for  the  morale  of  the  staff.  However,  receiving  feedback  with 
the  data  generated,  the  training  received,  the  support  from  the  higher  level  teams 
when  outbreaks  are  identified,  and  working  in  an  environment  with  support  and 
recognition  of  the  importance  of  the  tasks  performed  are  important  contributors  to 
the  motivation  of  the  reporting  staff. 

9.  The  surveillance  task  has  to  be  feasible  for  the  reporting  staff  considering  their 
skills,  resources  and  competing  tasks.  It  is  crucial  that  the  process  and  the  tool  used 
to  collect  and  analyze  the  data  are  simple  to  use,  flexible  to  changing  circumstances 
and  surveillance  needs,  and  standardized  to  all  surveillance  sites.  The  list  of  events 
has  to  be  prioritized  to  select  not  only  the  number  of  events  but  also  the  amount  of 
data  to  be  collected. 

10.  Finally,  a  fully  functioning  system  needs  to  use  the  information  generated  to  put  in 
place  public  health  measures  to  improve  the  health  of  the  population  under 
surveillance.  This  is  the  ultimate  motivator  for  reporting  staff  and  staff  at  the 
district,  regional  and  national  levels. 

Future  research 

This  study  developed  a  model  to  understand  the  key  considerations  when 
implementing  electronic  disease  surveillance  systems  in  developing  countries.  Due  to  the 
scarcity  of  published  literature  on  implementation  of  electronic  disease  surveillance 
systems,  this  effort  primarily  relied  on  qualitative  research  methodology.  The  next  step 
would  be  to  identify  and  create  quantitative  metrics  or  benchmarks  for  the  concepts  in  the 
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model  to  be  able  to  measure  the  probability  of  success  of  the  implementation  of  a 
surveillance  system  in  such  settings. 

This  model  focused  extensively  on  the  reporting  side  of  the  surveillance  system, 
as  in  developing  countries  the  main  concern  is  to  have  a  system  that  generates  data. 
However,  further  research  needs  to  explore  systems  where  there  is  a  greater  use  of  the 
data  at  the  national  and  regional  levels.  This  would  allow  obtaining  the  maximum  benefit 

;®l|i 

of  the  information  generated  and  develop  a  better  response  to  it. 

This  model  was  developed  using  data  from  civilian  and  military  institutions,  but 
identifying  implementation  factors  in  common  for  both  of  these  institutions  that  could  be 
extrapolated  to  other  institutions.  However,  it  would  be  appropriate  to  explore  the 
divergences  of  both  realities  to  develop  recommendations  that  are  specific  to  each  of 
them. 

Finally,  the  first  version  of  this  model  was  developed  based  on  the  data  collected 
in  Peru,  and  then  it  was  validated  by  literature  from  other  developing  countries,  as  well  as 
from  the  experience  of  experts  who  have  worked  in  several  developing  countries. 
Nevertheless,  it  would  be  important  to  apply  this  model  to  systems  in  countries  with 
resource  constraints  different  from  Peru,  in  order  to  fully  understand  the  applicability  of 
the  model. 
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SUMMARY  AND  CONCLUSION 


To  build  an  initial  model  to  understand  the  successful  implementation  of 
electronic  disease  surveillance  systems,  the  researcher  used  grounded  theory 
methodology  to  analyze  data  collected  from  disease  surveillance  stakeholders  from  the 
MOH  and  the  Armed  Forces,  as  well  as  from  foreign  organizations  that  supported  this 
process  in  Peru.  The  concepts  identified  in  the  model  were  triangulated  with  the  findings 
in  the  published  and  grey  literature  and  the  consensus  reached  by  subject  matter  experts 
on  a  workshop  organized  by  the  researcher  in  Washington,  DC.  The  final  revised  model 
is  presented  in  Figure  14. 

There  are  major  lessons  learned  for  electronic  disease  surveillance  system 
implementation,  monitoring  and  evaluation  from  this  research.  The  success  in  the 
implementation  of  these  systems  was  conceived  by  the  confluence  of  several  factors  at 
different  levels  of  stakeholders:  at  the  institutional  administrative  level,  at  the  national 
and  regional  disease  surveillance  team  level  and  at  the  surveillance  site  level. 

At  the  administrative  level,  the  key  element  to  start  this  process  was  the 
recognition  of  a  surveillance  need  in  the  organization  and  the  awareness  of  the 
significance  of  this  need.  The  administrative  level  will  then  commit  to  implement  and 
sustain  this  program  through  the  development  of  a  strategic  vision  and  plan,  the 
establishment  of  an  organizational  structure  of  the  system,  the  development  or  update  of 
surveillance  legislation,  and  the  assignment  of  financial,  technical  and  other  resources 
required  to  conduct  this  process.  An  external  organization  may  play  a  role  in  this  process 
by  providing  technical  and  additional  financial  assistance  and  if  that  is  the  case. 
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coordination  mechanisms  have  to  be  put  in  place  to  maximize  the  collaboration  between 
these  organizations. 
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Outcome:  Timely  and  quality  information 

t 


SURVEILLANCE  SITE 


s - 

\ 

Surveillance 

Feasibility 

Simplicity  Standardization 

Flexibility  Prioritization 

V _ _ _ 

y 

r 

R  esources 

1 

l 

Access  Availability 

J 

Data  analysis 


Surveillance  Team 
Skilled  Motivated 


r 

Administrative 

L 

support 

J 

|  Training  J  c 


Feedback 


|  Incentives  j 


Surveillance 

Awareness 


Support  j 

Monitoring 

Supervision 

11 

Accountability 


NATIONAL  AND  REGIONAL  SKILLED  SURVEILLANCE  TEAMS 


EXTERNAL 

ORGANIZATION 


Financial  support 


Technical  support 


Legislation 


Coordination 


Organizational  structure 


Strategic  vision  and  plan 


INSTITUTIONAL  ADMINISTRATIVE  LEVEL  COMMITMENT 


Figure  14:  Final  revised  model  for  implementation  of  electronic  disease  surveillance  systems  in  resource  limited  settings. 
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The  national  disease  surveillance  team,  often  in  collaboration  with  the  regional  teams, 
takes  charge  of  operationalizing  this  process.  This  team  should  be  composed  of  highly  motivated 
people  with  experience  in  disease  surveillance.  The  main  functions  for  these  teams  are  to  train 
the  reporting  staff;  monitor  the  timeliness  and  the  quality  of  the  data  generated  by  the  reporting 
staff;  and  give  feedback  with  the  analysis  of  the  data  collected  by  the  system  to  the 
administrative  level  and  the  surveillance  staff.  Supervision  was  also  found  important  in 
improving  the  quality  of  the  data  and  as  a  means  to  build  a  closer  relationship  with  the  reporting 
staff  and  strengthen  their  commitment.  The  resources  required  to  perform  these  functions  were 
critical. 

An  important  concept  in  the  model  is  the  availability  of  skilled  and  motivated  human 
resources  at  the  surveillance  sites.  Training  played  a  duplicative  role  in  providing  the  skills 
required  for  the  reporting  staff  to  fulfill  the  surveillance  tasks,  as  well  as  to  increase  the 
awareness  of  the  significance  of  this  task.  This  increased  awareness  was  considered  a  motivator 
by  itself.  In  addition,  motivation  was  increased  by:  having  the  resources  in  place  to  perform  the 
task  (i.e.  computers  or  phones);  the  support  from  the  director  of  the  site  who  facilitated  the  task 
(i.e.  allowing  the  use  of  the  only  computer  at  the  site);  the  constant  support  and  response  to  the 
events  reported  from  the  national  team  (i.e.  response  to  outbreaks  reported);  the  feedback 

V 

received  with  the  analysis  of  the  data  collected  (i.<v  monthly’  bulletins);  and  the  use  of  incentives, 
such  as  the  recognition  of  the  best  performing  sites  by  the  highest  authority  in  the  organization. 
One  final  concept  to  consider  at  this  level,  in  order  to  facilitate  the  reporting  staff  to  provide 
timely  and  high  quality  data,  is  the  feasibility  of  the  surveillance  task.  For  this  task  to  be  feasible, 
the  reporting  process  and  the  reporting  tool  should  be  simple  and  flexible,  and  the  materials  and 
guidelines  standardized  and  clear.  The  tool  can  be  designed  to  facilitate  the  reporting  for  the 
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surveillance  staff  and  the  monitoring  tasks  for  the  national  and  regional  teams.  Finally,  the  data 
generated  by  the  system  needs  to  guide  decision  makers  to  put  in  place  preventive  actions  for  the 
improvement  of  the  health  status  of  their  population.  To  enhance  the  health  of  their  population 
was  recognized  as  the  main  motivator  for  all  groups  of  disease  surveillance  stakeholders. 

It  is  noteworthy  that  the  first  review  of  the  literature  that  focused  on  the  implementation 
of  health  programs  in  developing  countries  identified  several  important  concepts  that  were 
included  in  the  final  model  developed  specifically  for  electronic  disease  surveillance  systems. 
Even  when  these  concepts  were  not  well  developed  in  the  literature,  they  served  as  an  initial 
guidance  and  reference.  Key  factors  identified  with  this  initial  literature  review  were:  political 
environment,  leadership,  legislation  in  place,  external  support,  monitoring,  supervision,  training, 
communication  and  engagement  of  stakeholder  groups,  program  flexibility,  human  resources  and 
other  resources,  motivation,  and  social  environment. 

The  second  review  of  the  literature,  which  specifically  targeted  electronic  disease 
surveillance  systems  in  developing  countries,  revealed  most  of  the  concepts  identified  in  our 
interviews.  Moreover,  the  consensus  reached  by  experts  on  this  field  on  the  key  factors  related  to 
the  successful  implementation,  provided  the  confidence  to  the  researcher  that  the  model 
developed  was  coherent  and  supported  by  other  similar  experiences.  However,  the  relationship 
between  those  concepts  could  not  be  assessed  and  fully  explored  by  these  methods.  The  use  of 
grounded  theory  methodology  proved  critical  for  this  purpose. 

Several  limitations  should  be  recognized  about  the  work  presented.  This  work  is 
qualitative  which  intrinsically  means  that  it  is  strongly  dependent  on  the  researcher’s  skills  and 
her  biases.  The  researcher  had  extensive  work  experience  on  disease  surveillance  implementation 
in  Peru  which  was  an  asset  to  understand  the  data,  but  at  the  same  time  could  bias  the  ideas  and 
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experiences  presented  by  research  participants.  To  be  able  to  prevent  this  or  correct  it,  the 
researcher  kept  a  personal  journal  to  document  the  research  process. 

An  additional  limitation  relates  to  the  sampling  methodology.  Due  to  regulatory 
requirements,  the  researcher  could  only  contact  the  potential  research  participants  twice  in 
recruitment  efforts.  This  restriction  prevented  interviewing  some  identified  stakeholders.  When 
this  situation  arose,  the  researcher  found  replacement  participants  from  the  same  stakeholder’s 
group  with  similar  characteristics  and  experience.  Due  to  logistic  limitations,  most  of  the 
interviews  took  place  in  the  capital  city.  However,  the  people  selected  to  be  interviewed  had 
worked  in  several  sites  across  the  country,  including  remote  areas,  allowing  the  researcher  to 
capture  this  information  as  well. 

Despite  the  limitations  described  above,  the  researcher  believes  this  is  the  first  step  for 
developing  a  model  with  the  ability  to  predict  successful  implementation  of  electronic  disease 
surveillance  systems  in  settings  with  resource  constraints.  This  exploratory  study  can  serve  as  the 
basis  for  future  research  to  test  the  model  in  other  realities  and  then  create  variables  that  can 
actually  be  quantified.  Being  able  to  predict  if  the  implementation  of  a  program  will  be 
successful  can  greatly  serve  for  better  allocation  and  use  of  resources.  Moreover,  such  a  model 
may  empower  decision  makers  from  governmental  and  non-governmental  organizations  that 
support  these  initiatives  and  be  able  to  identify  places  for  improvement  to  guarantee  success  in 
this  process.  Last,  the  researcher  believes  this  model  constitutes  an  initial  tool  to  understand 
program  implementation  of  other  health  related  programs  in  developing  countries. 
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Appendix  A:  Systematic  review  methodology 


The  researcher  and  her  thesis  advisor  (CM,  RWG)  developed  the  research  protocol 
followed  in  this  study.  The  protocol  and  subsequent  study  was  designed  using  the  PRISMA 
statement  format  (58;  67). 

To  be  included  in  this  systematic  review,  studies  had  to  meet  the  following  criteria:  I )  be 
indexed  to  one  of  the  selected  information  sources,  2)  be  written  in  English,  3)  be  focused  on  a 
developing  country(ies),  and  4)  must  have  addressed  the  implementation  of  a  specific  health- 
related  program.  There  were  no  restrictions  on  date  of  publication,  specific  research  methods, 
focus  of  intervention,  or  outcome  measure(s)  addressed.  Opinion  or  commentary  manuscripts 
were  excluded  only  if  they  did  not  meet  the  four  criteria  outlined  above. 

Studies  were  identified  by  searching  the  following  electronic  databases:  PubMed  (1957- 
2008),  embase®  (1974-2009),  CINAHL®  ( 1976-2009),  PsycINFO®  (1967-2009),  and  Current 
Contents  Connect®  (1975-2009).  The  last  search  was  conducted  on  25  March  2009.  The 
researcher  conducted  an  initial  search  using  the  following  key  word  scheme  (using  the  embase® 
example):  (('program  evaluation'/exp  OR  'program  evaluation')  OR  (’program  development’/exp 
OR  'program  development')  OR  ('pilot  projects'/exp  OR  'pilot  projects'))  AND  ((’health 
promotion'/exp  OR  health  promotion’)  OR  (’population  surveillance’/exp  OR  'population 
surveillance')  OR  (’health  education'/exp  OR  'health  education')  OR  ('health  policy'/exp  OR 
'health  policy’))  AND  ((’biomedical  technology'/exp  OR  'biomedical  technology')  OR 
('information  dissemination'/exp  OR  'information  dissemination')  OR  ('public  health 
informatics'/exp  OR  'public  health  informatics’))  AND  ((’developing  countries'/exp  OR 
'developing  countries')  OR  'global'  OR  'provincial'  OR  'rural'  OR  'regional'). 
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Upon  completion  of  the  first  search,  using  above  criteria,  the  researcher  and  her  advisor 
determined  that  the  initial  search  criteria  was  too  tight,  resulting  in  potential  loss  of  important 
studies.  The  researcher  conducted  a  second  revised  bibliographic  search  using  broader  key 
words:  ‘implementation’  AND  ‘global’  AND  ‘health’.  The  researcher  anticipated  that  the  revised 
search  criteria  would  yield  additional  studies  for  review;  many  without  a  focus  on  technology  or 
defined  program  structure.  The  potential  gain  of  new  information,  however,  outweighed  the  cost 
of  reviewing  additional  abstracts. 

Study  eligibility  assessment  within  each  bibliographic  search  was  performed 
independently  by  both  the  researcher  and  her  advisor  in  an  un-blinded  standardized  manner. 
Results  of  the  reviews  were  compared  and  disagreements  were  resolved  through  a  consensus 
process.  Specifically,  the  person  opting  for  inclusion  would  present  the  abstract  to  the  other 
person,  and  defend  why  the  manuscript  should  be  included.  Discussion  would  ensue  until 
consensus  was  reached  on  inclusion  or  exclusion  of  the  abstract. 

In  preparation  for  the  review,  the  researcher  developed  a  single  data  extraction  instrument 
(see  Appendix  B).  The  instrument  w'as  initially  pilot  tested  by  CM  and  RG  on  10  papers,  then 
revised  to  clarify  confusing  points  and  eliminate  unnecessary  elements.  The  researcher 
conducted  the  bibliographic  searches  using  the  identified  electronic  databases,  per  protocol,  and 
provided  a  copy  of  all  results  to  her  advisor  (RWG)  for  concurrent  review.  In  the  initial  stage  of 
abstract  review,  the  researcher  and  advisor  used  the  study  selection  criteria  as  a  basis  for 
determining  if  full  review  was  appropriate.  When  full  review  was  warranted,  the  researcher 
obtained  the  full-text  version  of  the  manuscripts  and  provided  a  copy  of  each  to  her  advisor.  CM 
and  RWG  extracted  data  from  each  full-text  manuscript  to  the  extraction  instrument,  assessed  its 
applicability,  and  made  an  independent  evaluation  of  its  inclusion  in  the  systematic  review.  Both 
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the  researcher  and  her  advisor  reviewed  their  extraction  forms  collectively,  and  came  to 
consensus  regarding  which  manuscripts  would  be  ultimately  included. 

The  data  extraction  form  included  specific  inclusion  criteria  and  descriptive  information 
on  the  study.  Key  dimensions  of  program  implementation  were  documented  qualitatively.  Basic 
qualitative  coding  methodology  was  applied  to  the  documented  dimensions  in  an  effort  to 
identify  key  themes  used  in  reporting  the  findings.  While  the  themes  were  identified  after 
completion  of  the  review,  the  following  data  items  were  adopted  from  the  work  of  Paul- 
Ebhohimhen  and  colleagues  (73)  and  served  as  the  basis  for  the  format  of  the  tables  (See 
Appendixes  C,  D,  E,  and  F).  They  include:  1)  reference,  2)  sample  characteristics,  3)  details  of 
intervention  (including  objective),  4)  study  design  and  analysis,  and  5)  important 
observations/considerations. 

Research  on  program  implementation  within  developing  countries  is  traditionally 
descriptive  in  nature  and  often  lacks  tight  controls  to  reduce  bias.  As  such,  the  researcher  did  not 
apply  specific  criteria  in  assessing  risk  of  bias  within  individual  studies. 
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Appendix  B:  Health-related  Program  Implementation  Systematic  Review  Abstract  Evaluation  Form 


First  author,  year  published: _ 

Basic  Inclusion  /  Exclusion  Criteria: _ 

Article  related  in  some  manner  to  program  implementation? 

_ Yes  _ No  [EXCLUDE] 

Written  in  English:  _ Yes  _ No  [EXCLUDE] 

Focused  on  implementation  in  developed  countries.? _ No  _ Yes 

[EXCLUDE] 

Type  of  article: 

_ Research  (including  education  research) 

_ Systematic  review 

_ Program  evaluation/quality  improvement  study 

_ Commentary/opinion  paper/perspectives  paper 

_ Education  paper  (describes  curriculum,  assessment,  etc.) 

_ Other  (specify: _ 

_ ) 

KEY  DIMENSIONS  OF  PROGRAM  IMPLEMENTATION  ADDRESSED  OR  STUDIED:  (List. 
If  none,  exclude) 

POSSIBLY  INCLUDE  STUDY  IN  SYSTEMATIC  REVIEW:  _ Yes  _  No 

If  not,  why  not? 


Setting: 

Number  of  Sites 
Number  of  district/provinces 
Number  of  Countries 
Number  of  regions 

Number  of  rural  sites 
Number  or  urban  sites 


Type  of  Setting: 

Community  or  family  based 
School-based 
Clinic/hospital  based 
Employer-based 


If  research,  type  of  study: 

_  Observational  descriptive 

_  Observational  analytic: 

_ Cross-sectional 

_ Cohort 

_ Case-control 

_ Ecologic 

_  Experimental  or  quasi-experimental 

_  Tool  development  and/or  validation 

_  Other  (specify: _ ) 

If  research,  data  collected: 

_  Chart  review  (paper  or  electronic) 

_  Survey/interview:  self-administered 

_  Survey/interview:  administered  in  person  by  interviewer 

_  Survey/interview:  administered  by  email  or  phone 

_  Participant  observation 

_  Focus  groups/community  forum 

_  Outcome  variables  measured  (specify: _ ) 

_  Other: _ 

If  systematic  review: 

Type: 

_ Summaries  or  descriptions 

_ Meta-analysis 

_ meta-synthesis 

Purpose: 

Number  of  papers  reviewed/included- 
Summary  of  findings: 

If  program  evaluation  or  quality  improvement, 

What  was  evaluated? 

What  is  the  "level"  of  the  evaluation  data? 

_ Patient  or  user  satisfaction 

_ Behavior  change 

_ Process 

_ Impact  (intermediate  outcomes) 

_ Long-term  outcomes 

_ Formative  evaluation 

_ Summative  evaluation 

_ other: _ 
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Appendix  C:  Summary  of  included  studies  related  to  Program  Design 


Reference 

Sample 

characteristics 

Details  of  intervention 

Important  observations 
(primary) 

Important  observations  (secondary) 

Ashroek  el 
al.  2005  (4) 

Governmental 
primary  health  care 
centers,  health 
posts  and  sub- 
health  posts  of 
Nawalparasi,  a 
rural  lowland 
district  in  Nepal. 

Practical  Approach  to  Lung 
Health  (PAL)-Nepal  is  an 
adaptation  of  generic  PAL 
WHO  guidelines. 

Objective :  to  improve 
syndromic  management  of 
lung  diseases  in  youths  >5 
yr  and  adults  of  middle-low 
income  countries.  Includes 
clinical  practice  guidelines 
and  training  materials. 
Implementation  strategy 
consisted  of  training 
primary  care  health  workers 
using  didactic  presentation 
and  interactive  exercises. 

Study  type: 

Qualitative 
assessment  of  the 
development  and 
implementation 
process.  Assessed 
three  components: 
guidelines,  training, 
and  context  of 
health  workers. 

Analysis:  Multi-rater 
kappa  analysis  and 
qualitative  analysis. 
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Program  design: 

1 )  Training  materials  of 
high  quality,  2)  Easily 
accessible,  comprehensive 
guideline  document,  3) 
Training  schedule  doesn’t 
compensate  for  turnover,  4) 
Algorithms  are  many  and 
complex,  5)  Training  not 
modified  to  match  worker 
qualifications/capabilities, 
and  6)  Training  alone  is  not 
effective. 

Engagement: 

1)  Stakeholder  involvement  (patients 
views  not  incorporated). 

Accountability/leadership: 

1)  Lack  of  recommendations  for 
monitoring  and  auditing. 

2)  Limited  recognition  of  human 
resource  barriers. 

Environment:  1 )  Powerful  stakeholders 
at  national  level  and  international  donor 
support,  2)  Positive  results  necessary  to 
marshal  political  will  and  financial 
resources,  and  3)  High  staff  turnover 
and  absenteeism. 

Valcnciano 
et  al.  2004 
(91) 


ALERT  initially  in 
4  pilot  districts 
then  expanded 
across  Serbia  over 
a  6-month  period. 
All  25  districts  and 
156  municipalities 
(98%). 


ALERT  is  a  communicable 
disease  surveillance  system 
that  relics  on  notification  of 
1 1  syndromes  by  primary 
care  facilities.  Data  are 
analyzed  weekly  at  district 
level  and  transmitted  to 
national  epidemiologists 
where  they  are 
computerized.  Public 
domain  software  used  for 
relational  data  entry  and 
production  of  interactive 
reports. 


Amaral  et  al.  3  of  9  states  in 
2005(2)  Northeast  region  in 

Brazil;  Inclusion 
criteria:  year  2000 
population  of 


5,000- 

50,()00;cxcluded 
larger  and  smaller 
municipalities;  144 
municipalities  (48 
in  each  group). 


Objective:  IMCI  (Integrated 
Management  of  Childhood 
Illness)  aims  to  reduce 
leading  causes  of  death 
among  young  children; 
contents:  case  management 
al  health  facilities, 
strengthening  health 
systems  support  and 
improving  family  & 
community  practices. 


1 2  countries 
restricted  to  those 
likely  to  implement 
IMCI  fully  in  large 


Objectives:  IMCI  strategy 
( 1 996-200 1  )  aims  to  reduce 
mortality  among  children 
under  the  age  of  5  years. 


Bryce  et  al. 
2005(13) 


Study  type : 
evaluation  using 
CDC  guidelines. 
Team  analyzed  the 
ALERT  database, 
revised  documents 
and  reports, 
conducted  structured 
interviews  with  key 
informants  al  all 
levels  and  held  one- 
day  evaluation 
workshop  with  all 
district 

epidemiologists. 

Analysis :  Varied  by 
attribute  under 
analysis. 

Study  type-. 
Observational  study 
comparing  3  groups 
1999-2002:  High 
IMCI  training 
coverage  >50-60% 
(23  munic.);  low 
(216  munic.);  no 
IMCI  (204  munic.); 
Health  facility 
survey:  23  high 
IMCI  munic.  (48 
facilities)  vs.  23  NO 
IMCI(48  facilities); 
Analysis:  ANOVA, 
analysis  of  trends. 
Study  type:  7  years 
of  country-based 
evaluation;  Impact 
model  developed  in 


Program  design:  Environment:  1)  Perception  of  ALERT 

1)  System  simple  (data  as  parallel  system  not  interoperable 

entry  takes  one  minute  per  with  routine  system,  and  2) 
week)  and  flexible,  2)  Acceptability  higher  at  national  level 

Using  syndromic  case  because  of  timely  reception  of  data, 

definitions  allows  reporting 

without  lab  confirmation.  Accountability/leadersliip:  1)  Reporting 

3)  Uses  standardized  data  at  procedures  not  incorporated  in  the 
national  level, 4)  Lack  of  public  health  laws  and  regulations. 

ALERT  connection  with 
individual  control  measures, 

5)  Strengthened  data  Engagement: 

analysis  through  training.  1 )  Program  improved  communication 

between  all  surveillance  actors  and 
strengthened  surveillance  network,  2) 
Improved  feedback  and  communication 
at  republic  and  district  level. 


Program  design:  Environment: 

1 )  Training  component  1 )  Experienced  high  staff  turnover, 

limited  to  physicians  and 

nurses,  Engagement: 

2)  Lack  of  association  I )  Program  improved  quality  of  care  in 

between  IMCI  training  IMCI  facilities. 

coverage  and  infant 

mortality  after  controlling  Accountability/leadership: 

for  confounding  variables.  I  Supervision  was  poor 


Program  design:  Environment: 

I )  Technical  guidelines  on  1 )  Staff  turnover  an  impediment, 
delivering  interventions  at 

family  and  community  AccountabilitsZleadership : 
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geographical  areas, 
within  the  2  years 
after  the 
assessment  visit 
(impact  evaluation 
period  of  2-3 
years). 

Implemented  in  countries 
with  moderate  to  high 
levels  of  child  mortality  in 

3  phases:  1)  introductory 
phase:  orientation  meetings, 
trained  key  decision 
makers,  management 
structure,  2)  early 
implementation  phase: 
limited  geographic  areas 
and  plan  for  expansion  3) 
expansion  phase. 

1999-2000.  3  types 
of  studies  included: 

1)  12  country  in- 
depth  assessments 
of  1  MCI 

implementation;  2) 
Compatible  designs 
in  Bangladesh,  Peru, 
Tanzania  &  Uganda; 
each  tailored  to  the 
stage  and 
characteristics;  3) 
Cross-site  analyses 
of  specific  activities. 
Studies  were 
prospective, 
retrospective  and 
mixed  designs. 
Analysis:  Compared 
findings  of  MCE- 
IMCI  relative  to 
program 
expectations. 

levels  were  slow  to  appear, 

2)  Training  courses  required 
substantial  organizational 
skill  and  dedicated  staff,  3) 
Integrated  guidelines 
supported  by  high  quality 
training  and  supportive 
supervision. 

1)  Evaluation  not  weighted  for  health 
system  limitations  (political 
commitment,  human  resources, 
financing,  integrated  or  coordinated 
program  management,  effective 
decentralization). 

Forster  et  al. 
2008 (37) 

10  of  21  programs 
completed 
questionnaire  on 
EMR  systems, 
resources,  reporting 
systems,  data 
storage,  quality 
control  measures 
and  tracing  of 
patients  lost  to 
follow  up;  21  sites 
surveyed;  Africa 
(18),  Asia  (2), 

South  America  ( 1), 

Antiretroviral  Treatment  in 
Lower  Income  Countries 
(ART-L1NC)  collaboration; 
a  network  of  treatment  sites 
of  International 
Epidemiological  Databases 
to  Evaluate  AIDS  (IeDEA). 

Study  type:  Survey 
used  to  create 
quality  indicators. 
Analysis:  Factor 
analysis;  Reliability 
tests;  Site  data  and 
quality  compared 
using  univariate  and 
multivariate 
maximum  likelihood 
logit  models; 
models  for  clustered 
data  were  used  to 
evaluate  predictors. 

Program  design: 

1 )  Training  of  staff  and 
clerk  hours  spent  per  week 
per  100  patients  on  ART 
were  associated  with  a 
decreased  likelihood  of 
missing  data,  2)  Staff 
inadequately  trained  to 
manage  data  and  trace 
patients  lost  to  follow  up,  3) 
Limited  number  of  database 
features,  4)  Lack  of 
standardized  coding  or 
controlled  vocabularies,  5) 
Advantage  of  central  server 
over  local  server  is  stability 
and  security  BUT  raised 

Engagement: 

1)  Need  development  of  affordable  and 
sustainable  solutions  to  support  clinical 
care, 
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Herbst  et  a). 
1999 (44) 

1  hospital  in  the 
Northern  Province 
of  South  Africa  - 
pilot  site;  42 
hospitals  total. 

Implementation  of  a 

Hospital  Information 

System  (HIS)  in  42 
hospitals  in  Northern 
Province,  South  Africa. 

Began  with  one  pilot  site. 
Each  hospital  had  its  own 
application  sever  managing 
local  data,  but  shared  some 
patient-level  data  with  other 
hospitals  and  to  central 
server. 

Study  type: 
Formative 
evaluation  program 
to  assess  a  new 
information 
initiative  in  South 
Africa. 

Analysis:  not 
addressed 

Huicho  ct  al. 
2005  (48) 

34  health  districts 
(DISAs)  in  24 
departments  in 

Peru  (1996-2000). 

IMCI  strategy  depicts  3 
components:  health  system 
improvements,  health 
worker  training  and 
community  level 
interventions. 

Study  type: 
Ecological  study. 

Analysis:  Bivariate 
analysis  of  key 
variables. 

Okubagzhi  et 
al.  2002  (70) 

Ethiopia  National 
project 

Ethiopia  Multi-sectoral 
HIV/AIDS  Project 
(EMSAP)  finances  a  3-year 
government  strategic  plan. 
Objective:  To  reduce  the 
spread  of  epidemic, 
alleviate  impact  and 
increase  access  to 
treatment,  care  and  support 
for  those  infected  and 
affected. 

Study  type:  not 
addressed 

Analysis:  not 
addressed 
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issues  of  data  ownership,  6) 

Open  resource  databases 
easily  adapted  to  local 
requirements,  7)  Several 

strategies  to  trace  patients. _ 

Program  design:  Engagement: 

1 )  Training  for  software  use  I )  Enthusiasm  and  understanding  of 

by  end-users  effective  and  need  for  the  systems  at  senior-level 

2)  Interventions  should  be  needs  to  be  diffused  down  to  managers 

sensitive  to  the  changes  of  and  clinicians  at  all  levels, 
political,  professional  and 

managerial  environments. 


Program  design:  Environment: 

1 )  Correlation  between  I )  High  turnover  of  trained  staff, 

training  coverage, 

outpatient  utilization  and 
vaccine  coverage  were 
weak  and  non-significant 

2)  No  significant 
associations  between 
clinical  training  and 
mortality  or  nutrition 

indicators. _ 

Program  design:  Environment: 

1)  Manuals  proven  useful  I )  Brief  time  to  internalize 
instruments  but  frequent  implementation  process,  2) 
interpretation  required,  2)  International  partnership  instrumental 
Conducted  pilot  phase  with  in  creating  government  interest  to  take 
limited  number  of  districts,  urgent  action,  and  3)  Government 

3)  Manuals  and  training  is  established  a  comprehensive  policy  and 
NOT  sufficient,  4)  Complex  strategic  plan. 

design  was  barrier  to  Engagement: 

coordination.  I )  Encouraged  interventions  at  a 

community  level  by  allowing  funds 
to  flow  directly  to  communities. 


Appendix  D:  Summary  of  included  studies  related  to  accountability  /  leadership 


Reference 

Sample 

characteristics 

Details  of 

intervention 

Study  design  and 
analysis 

Important  observations 
(primary) 

Important  observations  (secondary) 

Freedman 

et  al.  2(307 
(39) 

Not  addressed 

Outlined  four  major 
global  safe 
motherhood 

initiatives:  1) 

Averting  Maternal 
Death  and  Disability 
(AMDD),  2) 

Immpact,  3)  the 
Skilled  Care  Initiate 
(SCI)  and  4) 

ACCESS. 

Study  type: 

Description  of  field 
experiences  that 
provide  a  set  of 
central  lessons- 

learned. 

Analysis:  not 
addressed 

Accountabil  ity/leade  rsh  ip: 

1 )  Implementation  efforts 
should  not  wail  until  all 
central  issues  resolved,  2) 
Establish  legal  and 
regulatory  mechanisms  to 
ensure  accountability,  3) 
Define  a  small  number  of 

indicators  that  will  not 
overwhelm  fragile 
reporting  systems. 

Environment: 

1)  Infrastructure  support  of  health  system  important,  2) 
Donor  support  critical,  3)  Positive  social  atmosphere 
created  by  local  chiefs  and  4)  Staff  al  all  le  vels  viewed 
themselves  as  part  of  a  team. 

Program  design: 

1)  Program  should  be  based  on  evidence  and  on  relevant 
information  about  local  context. 

Engagement: 

1)  Comprehensive  framework  for  recruiting,  developing 
and  retaining  workforce  important,  and  2)  context 
specific  information  about  motivating  factors  for  service 
quality  and  use. 

Khatri  el  al. 
2002 (57) 

National  TB 

control 
program  in 

India  where 

health 

infrastructure 

present. 

DOTS  strategy 
expanded  rapidly 
since  late  1998  and 

in  both  2000  and 

2001  the  country 
probably  accounted 
for  more  than  half 
the  global  increase  in 
the  number  of 
patients  treated 
under  DOTS. 

Study  type: 
Commentary  piece. 

Analysis:  not 
addressed 

Aceountability/leadersbip: 
I)  Program  orientation 
towards  problem  solving, 
and  accountability 
important,  2)  Regular 
monitoring  and  timely 
feedback  critical  and 
should  be  tied  to  funding, 
3),  Supervision  al  all 
levels  essential,  and  4) 
Requires  specialized 
supervisory  staff  who  can 
promote  accountability. 

Program  design: 

I)  All  technical  policies  and  detailed  training  modules 
written  and  field  tested,  2)  Phased  program  expansion  is 
essential,  3)  Focus  areas  must  be  carefully  selected  and 
maintained. 

Engagement: 

1 )  Regular  interaction  among  all  levels  of  staff  led  to 
large  body  of  highly  skilled  and  motivated  workers, 

2)  Required  building  government  commitment  to 
program. 

Environment: 

1)  Adequate  funding  required  as  is  flexibility  of  use. 
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Hotchkiss 

Country  of 

Program  to 

Study  type: 

ctal.  2006 

Georgia.  Center 

standardize  and 

Longitudinal  pre-post 

(46) 

for  Public 

facilitate  analysis  of 

quasi  experimental 

Health  (CPH) 

vaccine  preventable 

research  design, 

office  from  12 

disease  (VPD)  data 

supplemented  with 

rayons  (or 

and  its  translation 

data  from 

districts)  in 

into  public  health 

surveillance  system 

Region  of 

actions.  Intervention 

record  reviews  and 

Imereti 

components 

focus  group 

(intervention) 

included:  (1) 

discussions. 

and  3  rayons 

Surveillance 

(control) 

guidelines  for  rayon- 

Analysis:  Regression 

participated  in 

level  public  health 

modeling  to  assess 

study. 

managers  with 

the  effect  of 

surveillance 

intervention  package 

handbook  for  health 
facility  staff,  (2)  Job 
aid  for  rayon  CPH 
offices  to  help  guide 
analysis  and  use  of 
VPD  data,  3) 

Training  in  the 
guidelines  for  both 
rayon  CPH  and 
facility  staff,  4)  On- 
the-job  technical 
assistance,  and  5 ) 
Financial 

reimbursement 
system  for  specified 
analysis  and 
response  activities. 

on  selected  outcomes. 

Cults  et  al. 

Mozambique 

Project  objectives 

Study  type: 

Accountability/leadership: 

1 )  Lack  of  CPH  authority 
to  impose  penalties  on 
low-performing  health 
facilities,  2)  Weak 
accountability 
relationships  within 
Georgia's  health  system, 
and  3)  Lack  of  clarity 
about  which  level  bears 
responsibility  for  key 
surveillance  and  response 
functions. 

Engagement: 

1 )  Poor  reporting  of  data  from  some  private  providers,  2) 
Program  led  to  an  increased  ability  of  CPH  staff  to  carry 
out  critical  surveillance  functions. 

Environment: 

1 )  Lack  of  telephones  and  electricity  in  health  facility  and 
CPH  offices  was  problematic,  2)  Generally  low  levels  of 
health  care  utilization  in  country,  3)  Limitations  of 
government  resources  to  carry  out  surveillance  and 
response. 

Accountability /leader ship:  Program  design: 


1 988  (25)  MOH  estimated  were  to  achieve  70%  Evaluation  paper. 

priority  target  coverage  of  infants  Monitoring  activities 
population  of  for  all  EPI  (Global  provided  useful 

24,000  children  Expanded  qualitative 

aged  0-1 1  Programme  on  information, 

months  and  Immunization) 

27,000  vaccines  and  of  Baseline  surveys 

pregnant  pregnant  women  for  were  also  performed, 

women.  tetanus  toxoid.  Main 

action  points:  Analysis:  not 

-Simple  uniform  addressed 

immunization 
schedule 

-Outreach  visits  in  3 
consecutive  monthly 
pulses. 

-Effective 
communications 
systems  to  inform 
villagers 

-Integrating  EPI  with 
MCH  (maternal  and 
child  health  care). 

-Making  best  use  of 
contacts  with  health 
services. 


1 )  Good  managerial 
ability, 

2)  Supervisory  needs  are 
high, 

3)  Continuous 
participatory  process  of 
monitoring  and 
evaluation, 

4)  Definition  of  specific 
roles  and  concrete 
tasks, 

5)  Commitment  of  senior 
personnel  for 
accountability. 


1)  Training  needs  are  high,  2)  All  mid-level  health 
workers  should  be  trained  to  do  tasks,  3)  Printed 
materials  are  helpful,  4)  Pilot  phase  is  useful,  5) 
Different  techniques  used  at  different  stages  of 
program  development. 

Engagement: 

I)  Working  with  people  at  grassroots  levels,  2)  Establish 
direct  contact  with  peripheral  level,  3)  To  show 
results  at  a  small  scale  can  be  used  as  motivator,  4) 
Senior  personnel  involvement  in  evaluation  and 
providing  feedback  can  be  used  as  motivator,  5)  Face 
to  face  communications  strategy  was  developed,  6) 
Plan  was  prepared  by  MOH  and  donor  agency  and 
discussed  with  provincial  authorities. 

Environment 

1)  Sufficient  human  and  material  resources, 

2)  Positive  attitudes  of  health  staff  and  aim  population. 
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Appendix  E:  Summary  of  included  studies  related  to  engagement 


Reference 

Sample 

Details  of  intervention 

Study  design  and 

Important  observations 

Important  observations  (secondary) 

characteristics 

analysis 

(primary) 

Albert  et  al. 

28  developing 

Objective:  Implement 

Study  type :  Cross- 

Engagement: 

Program  design: 

2007  ( 1 ) 

countries  selected 

food-based  dietary 

sectional:  survey  sent 

1 )  Used  well 

1 )  Lack  of  data  can  delay  the 

from  42  countries 

guidelines  (FBDG)  -  to 

by  email  (messages 

established  channels  of 

development  of  guidelines.  2) 

whose  government 

advise  the  general  public 

contained, 

communication  and 

Flexibility  and  creativity  important  to 

staff  participated  in 

on  nutritional  information 

organizations 

existing  training 

overcome  limited  resource  constraints. 

previous  workshop. 

tailored  for  each  country. 

involved,  use  of 

activities  to  promote 

4  case  studies:  India, 

Includes  policy  tool  used  to 

FBDG,  evaluation); 

FBDG,  2)  Involvement 

Environment: 

South  Africa,  Chile 

coordinate  nutrition 

Case  studies:  facc-to- 

of  all  relevant 

1 )  Required  political  support  and 

and  Guatemala 

programs  and  to  evaluate 

face  interviews. 

stakeholders  at  the 

coordination  of  efforts. 

(countries  with 

adequacy  of  the  food 

telephone  interviews, 

earliest  stages  critical. 

highly  diverse 

supply  in  meeting  the 

surveys  and  review  of 

Accountability/leadership: 

populations  and  a 

nutritional  needs  of  the 

documentation. 

1)  Process  evaluation  should  consider 

range  of  challenging 

population. 

personal  in-depth 

the  environment  and  potential  for 

nutrition  education 

knowledge  of 

change. 

needs). 

processes  followed. 
Analysis:  Not 
addressed 

De  Salazar  el 

A  municipality  of 

School-based  behavioral 

Study  type:  Not 

Engagement: 

Environment: 

al.  2005  (27) 

Cali  that  covers  4 

risk  factor  surveillance 

addressed 

1)  Empowered  local 

1)  Successfully  utilized  available 

schools  with  1500 

system  (SIVEA)  developed 

Analysis:  Not 

people  and 

resources  and  infrastructure  (including 

children,  followed 

by  Universidad  del  Valle, 

addressed 

communities  to  manage 

daily  activity)  at  school. 

by  expansion  to  7 

Colombia. 

their  own  system,  and 

educational 

2)  Applied  effective 

Accountability/leadership: 

institutions  reaching 

strategies  to  increase 

I)  Advocacy,  communication  and 

1300  children 

key  stakeholder 

advertising  used  to  involve  and 

participation  and 

motivate  parlies  within  and  outside 

political  will. 

school. 
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Program  design: 

1 )  Surveillance  results  were  used  in 

decision  making. 

EANMAT 
2001 (31) 

Kenya,  Tanzania, 
Uganda  and 

Rwanda;  all 
countries  have  8 

official  sentinel  sites 
(SS)  except  Rwanda 
(4). 

Important  selection 
criteria  included: 

transmission 
intensity,  population 
density,  accessibility 
of  health  center, 
national  borders, 
geographical  spread, 
all  for  a  defined 

catchment  area  and 
stable  population 

EANMAT(East  African 
Network  for  Monitoring 
Antimalarial  Treatment)  to 
include: 

Intervention  included 

support 

documentation/goals, 
standardization  of  test 
methodology  and  field 
manual,  number  and 
location  of  SS,  monitoring 
of  treatment,  computer  data 
entry  program, 
internal  evaluation, 
appointment  of  network 
manager,  and  policy, 

Each  national  team 
comprised  8  staff  (4 
clinical,  4  parasitologists) 
trained  in  WHO 
methodology.  Every  2 
years,  meetings  to  discuss 
policy  implications  of  data 

Study  type:  Not 
addressed 

Analysis:  Not 
addressed 

Engagement: 

1 )  Central  role  for 

MOH  regarding 
ownership, 
implementation  and 
political  support,  2) 
EANMAT  partnership 
with  MOH  contributed 

to  beneficial  attitude 
changes  and  impact  on 
development  of 
policies. 

Program  design: 

1 )  Network  should  be  built  in  stages 
(bottom-up  staged  development). 

Environment: 

1 )  High  staff  turnover,  2)  Resources 
and  data  shared  through  a  small 
network  (3-5  countries). 

Accountability/leadership: 

1 )  Treatment  efficacy  data  used  in 
evaluation  and  modification  of 
antimalarial  policies, 

2)  Coordinating  secretarial  essential 
for  project  organization  and  to  provide 
a  subregional,  malaria-specific 
scientific  focus. 

Jahan  et  al. 
2000 (5 1 ) 

Community  in 
Chiltagoing,  the 
coastal  area  of 

south-east 

Bangladesh. 

Community  health 
education  pilot  project 
tilled  Sanitation  and  Family 
Education  (SAFE). 

Study  type:  Cross 
sectional  qualitative 
survey. 

Analysis:  not 
addressed 

Engagement: 

1 )  Development, 
testing  and  refinement 
of  tailored  education 
messages  that  target  all 
members  of  a 
community  including 
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Lonrothetal.  4  Public-Private  Mix  DOTS  strategy. 


2004  (60) 


(PPM)  projects  were 
established  for 
DOTS 

implementation  in 
New  Dchli  and 
Pune.  India;  Ho  Chi 
Minh  City,  Viet 
Nam;  and  Nairobi, 
Kenya. 


Educational  activities  for 
PPs  were  similar  across 
projects  and  strongly 
emphasized  adopting 
WHO  recommended 
diagnostic  and  case 
management  principles. 


Study  type',  evaluation 
study.  Projects 
evaluated  within  a 
framework  developed 
by  WHO  with  the  aim 
of  enabling  cross¬ 
project  analysis; 
Analysis:  qualitative 
analyses  of  interview 
and  written  project 
documentation. 


men,  women  and 
children;  2)  Interactive 
communication  within 
the  community  led  to 
skills  development  and 
local  ownership  of  the 
program. 


Engagement: 

1 )  Access  to  diagnostic 
facilities,  educational 
activities  and 
supervision  were 
incentives  to  encourage 
participation,  2)  Direct 
financial  incentives 
were  NOT  associated 
with  improved 
diagnosis  or  treatment. 


Accountability/leadership: 

1 )  Improved  referrals  and  information 
systems  via  simple  practical  tools, 

2)  Stakeholders  engaged  in  active 
dialogue  beginning  18  months  before 
the  project  was  launched  and 
throughout  the  project. 

Program  design: 

I )  Generic  model  should  be  adaptable 
to  local  conditions. 


Matsudo  el  al  Community  in  Sao 
2003  (63)  Paulo. 


Agita  Sao  Paulo  promoted 
CDC  recommendations  of 
performing  at  least  30 
minutes  of  moderate- 
intensity  physical  activity 
per  day,  on  one  or  more 
sessions  each  day  on  most 
days  of  the  week.  Three 
specific  target  groups: 
students,  workers,  and  the 
elderly. 


Study  type:  Evaluation 
to  assess  program 
impact,  used  various 
surveys  to  assess 
changes  in  the  level 
of  physical  activity 
and  participant 
knowledge. 

Analysis:  Varied  by 
intervention 


Engagement: 

I )  Used  extensive  free 
media  coverage 


Environment: 

1 )  Collaboration  between  professional 
associations  and  government  sector 
important. 


Program  design: 

I )  Adopted  the  "two  hats”  approach, 
using  either  governmental  or 
nongovernmental  methodologies  to 
promote  physical  activity. 
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Nelcsone  et  One  hospital  and  Outbreak  surveillance 

al.  2006  (69)  seven  island  clinics  concept  developed  in 

in  Tuvalu  (Pacific  South  Africa,  adapted  and 
country).  transferred  to  a  Pacific 

island  nation 

(decentralized  model  with 
limited  number  of 
syndromes). 


Plamondon  Nicaragua's  Global  Tuberculosis  component  of 

et  al.  2008  Fund  project  (GFP)  Nicaragua's  Global  Fund 

(78)  project  (GFP).  Focus  on 

training  volunteer  health 
workers  and  establishing 
TB  Clubs  to  highlight 
potential  for  and  threats  to 
building  sustainable 
program  capacity. 

19  participants  included 
persons  affected  by  TB, 
brigadislas,  front-line 
health  workers,  regional 
and  national  staff,  and 
advisors.  Focus  groups 
with  6  additional 
participants. 


Zvavamwe  et  40  nurses  involved  Community-based  TB 

al  2008  (106)  in  providing  program, 

treatment  and  care 


Study  type: 

Evaluation  of  disease 
surveillance  system: 
zero  reporting 
progress, 
completeness  of 
reporting;  structured 
system  review  and  in- 
depth  interviews. 
Analysis:  not 
addressed 

Engagement: 

1 )  A  greater 
commitment  of 

resources  for  control 
strategics  and 
infrastructure 
improvements  after 
outbreak;  2)  No  formal 
feedback  to  clinic  staff 
on  their  performance. 

Accountability /leadership: 

1)  Need  for  ongoing  monitoring  and 
responsive  adaptation  to  feedback 
from  surveillance  agents  based  in  the 
periphery  of  the  health  system. 

Study  type: 

Engagement: 

Accountability/leadersliip: 

Ethnology-based  case 

1 )  Enhancing  capacity 

1)  Dependency  on  local  leadership 

study  design. 

among  health  workers. 

without  investment  in  leadership 

Explores  local 

2)  Providing  funding 

development  was  problematic,  2)  Lack 

experiences  from  the 

was  counterproductive 

of  prioritization  of  TB  control  program 

time  of  proposal 

because  it  established 

within  broader  public  health  system, 

development  through 

an  unsustainable 

and  3)  Gaps  in  evaluation  in  area  of 

to  performance- based 

precedent  of  incentives 

quality. 

evaluation.  Data 

for  participation,  3) 

collection  included: 

lack  of  worker 

Program  design: 

review  of  historical 

acknowledgement  a 

1 )  Lack  of  regular  follow-up  training 

documents,  secondary 

barrier  to  program 

or  education  materials. 

data  analysis, 

sustainment,  and  4) 

participant 

Improved 

observation,  semi- 

communication  al 

structured  and  in- 

local,  regional  and 

depth  interviews,  and 

national  levels 

focus  groups. 

Analysis:  iterative 

process 

Study  type: 

Engagement: 

Accountability/leadersliip: 

Descriptive 

1 )  Use  of  incentives 

1 )  Experienced  poor  supervision  of 

quantitative  study 

for  TB  patient  &  staff 

staff  and  inadequate  record  keeping. 
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of  TB  patients  in  the  using  structured  participation,  and  2) 

Omaheke  region  of  interviews.  Established  community 

Namibia  .  Analysis:  descriptive  based  structures  to 

statistics  support  TB  work. 

Dicketal.  Pilot  project  on  21  Farm  dwellers  were  Study  type:  Series  of  Engagement:  Program  design: 

2007(29)  farms  in  Boland  assisted  in  selecting  discrete  studies  I )  Perception  1)  LHWs  were  trained  in  other  useful 

Health  District,  suitable  peers  to  be  trained  including  1)  un-  developed  that  money  skills  to  deal  with  health  issues. 

Western  Cape  as  Lay  Health  Workers  blinded  cluster  and  time  were  saved 

Province,  South  (LHWs),  92%  were  female,  randomized  trial,  2)  through  program,  2) 

Africa,  during  1993-  formal  cost  study  Changed  LPfW  own 

95.  Phase  1:  Preparation:  (control  vs.  behavior,  became  "role 

Informing  each  farming  intervention),  and  3)  model  in  community," 

Randomized  trial  community  of  study,  TB  in-depth  interviewing,  and  3)  Led  to  decrease 

sample  size  of  21 1  and  selection  process  for  focus  groups,  field  dependence  on  external 

was  achieved  (106  LHWs,  done  before  notes  and  participant  sources  including  farm 

intervention  and  105  randomization.  Phase  2:  observation  to  management  and 

control  group)  that  Training  the  LHWs  with  5  describe  perceptions  health  services, 

commenced  TB  one-week  modules  offered  of  key  stakeholders, 

treatment  (2000  to  during  5  month  period  on  a 
2001).  decentralized  basis  in  Analysis:  Statistical 

farming  district,  comparison  of  groups, 

experiential  learning  nonparametric  testing, 

approach  was  used,  and  qualitative 

flexible  to  respond  to  methodology  (i.e. 

specific  needs  of  each  content  analysis  and 

learner,  each  group  6-10  coding), 

participants.  Phase  3: 
support  of  LHWs  with 
monthly  visits  during 
implementation. 

Researchers  fed  study 
findings  to  participants  for 
validation,  facilitated  by 
development  of  posters 


207 


yp) 

displayed  in  district  health 
facilities,  and  brochure  for 
policymakers  and 
community  members. 

Ryman  et  al. 

Review  of  the 

Studies  and  projects 

Study  type: 

Engagement: 

Program  design: 

2008  (83) 

published  and  gray 

conducted  to  improve 

Systematic  literature 

1 )  Included  food 

1 )  Focus  on  bringing  immunizations 

literature.  Initial 

routine  immunization 

review:  online 

incentives  for 

closer  to  the  community  using  non- 

inclusion  criteria 
applied  to  60  papers. 

programs  among  humans  in 
low-  or  middle-  income 

databases  search 
using  42  terms  for 

participation,  2) 
Improved  quality  of 

health  workers. 

Standardized  quality 
assessment  of 
methodology  of  25 
papers. 

countries. 

papers  in  English, 
French  or  Spanish 
from  1975-  2004. 

Gray  literature  was 
searched  on  35 

websites. 

Analysis:  Following 
abstract  review, 
papers  classified  in  3 
methodological 
groups  and  rated 
based  on  a 

standardized 

health  facility  practices 
can  improve 
immunization  rates, 
and  3)  Used  mass 
communication 
campaign/  information 
dissemination  to 

increase  demand  for 

vaccination 

Accountability/leadership: 

1 )  Tools  developed  to  assist 
community  workers  in  tracking  their 
home  visits. 

assessment. 
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Appendix  F:  Summary  of  included  studies  related  to  the  environment 


Reference 

Sample 

characteristics 

Details  of  intervention 

Study  design  and 
analysis 

Important  observations 
(primary) 

Important  observations  (secondary) 

Baral  el  al 
2006 (5) 

4  countries:  Brazil, 
Eritrea,  India  and 
Vietnam  that 
successfully  reduced 
malaria  morbidity. 

(if  U ' 

Brazil:  1989  Project  for 

Control  of  Malaria  in  Amazon 
Basin  (PCMAM);  Eritrea: 

1999  MOH  Program  for 

Control  of  HIV,  Malaria, 

STDs  and  TB  (HAMSET), 

India:  Enhanced  Malaria 

Control  Project  (EMCP); 
Vietnam:  MOH  focused  on 
malaria  as  priority. 

Study  type:  Cross- 
sectional:  review  of 
published  and 
unpublished  reports; 
interview  of  selected 
program  and  partner 
staff  involved  in  design, 
implementation  and/or 
supervision. 

Analysis:  not  addressed 

Environment: 

1)  Capacity  at  national  and 
sub-national  levels  (skilled 
technical  staff  at  sub¬ 
national  level),  2)  Sufficient 
and  flexible  financing  from 
government  and  partners,  3) 
Hands-on  technical  and 
programmatic  support  from 
partner  agencies  and  4) 
Decentralized  control  of 

resources. 

Program  design: 

1)  Includes  data-driven  decision 
making. 

A  ecoun  tabilit)’/l  eade  rsh  ip : 

1)  Active  leadership  at  all  levels  of 
government . 

Creali  el  al. 
2007  (22) 

Southeast  Sulawesi 

and  Maluku  in 

Indonesia  where 

97%  of  mothers 
gave  birth  at  home. 
839  mothers  with 
children  <  2  years 
old. 

Healthy  Mothers-Healthy 

Babies  Project  aims  to 
integrate  the  administration  of 
birth  dose  Hepatitis  B  vaccine 
into  an  expanded  routine 
delivery  care  package 
provided  by  village  midwives. 

Study  type:  Evaluation 
study  following  5  years 
of  implementation. 
Analysis:  descriptive 

Environment: 

1 )  Low  coverage  due  to 
health  system  structure 
rather  than  a  result  of 
community  practices  or  lack 
of  community  acceptance, 

2)  Mistrust  of  staff  who  are 
not  normally  part  of 
program,  3)  Reluctance  on 
the  part  of  immunization 
staff  to  allow  vaccine 
outside  of  cold  chain,  and  4) 
Difficulties  in  reliably  of 
supplying  potent  vaccine  to 
community  level  due  to 
limited  transportation 
system. 

Program  design: 

1)  Training  allowed  better 
understanding  of  roles  and  empowered 
village  midwives. 

Accountability/leadership: 

1)  Lack  of  clear  delineation  of 
responsibility  between  health  care 
professionals. 
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Deen  et  al. 

Hospitals  in 

Pediatrician  visits  each  study 

2003 (28) 

developing  countries 

hospital  3  times,  initially  for  2 

meeting  inclusion 

weeks,  then  for  a  week  3-6 

criteria. 

months  later  and  finally  for 
another  week  al  the  end  of 

Postal  survey  sent  to 
36  recommended 

study  year. 

hospitals;  23 

During  each  of  the  visits,  the 

(63.8%)  completed 

management  of  severely 

and  returned.  Pre- 

malnourished  children  is 

selection  visits 

evaluated  using  a  structured 

conducted  in  4 
hospitals  in  Africa 
and  implementation 
started  in  2 
hospitals. 

audit  instrument. 

Jacucci  ct 

New  Hope  hospital, 

South  Africa  launched  the 

al.  2006 

with  its  functioning 

creation  of  a  unified  health 

(50) 

health  information 

information  system  (HIS),  to 

system,  selected 

support  data  collection. 

from  group  of 

aggregation  and  analysis  al  the 

hospitals  in  Eastern 

district  level.  As  part  of 

Cape,  South  Africa. 

implementation,  hospitals 
were  provided  with  new  forms 
and  registers  for  data 
collection,  computers,  and  the 
HIS  software.  Training 
sessions  were  organized  for 
the  hospital  staff. 

Study  Type:  Initial 
feedback  from  extended 
field  test  of  the  WHO 
severe  malnutrition 
guidelines. 

Analysis',  not  addressed 


Study  type:  Case  study; 
Interviews  with  the 
hospital  manager,  the 
matron,  the  information 
officer  and  the  nurses  in 
the  wards.  Observed 
budget  review  meeting. 
Analysis:  descriptive 
qualitative  methods 


En  vi  ronment:  Engagement: 

I )  Health  worker  attitude  of  1 )  Lack  of  interest  or  motivation  of 
helplessness  or  not  being  rotating  junior  doctors, 

able  to  make  a  difference,  2) 

Lack  of  trust  in  guidelines,  Accountability/leadership: 

3)  Difficult  to  change  I )  Lack  of  awareness  of  shortcoming 

established  practices,  and  4)  in  management  of  severe  malnutrition, 
Existing  hospital  policies,  and  2)  Monitoring  integrated  into  the 
structure  and  politics  routine, 

impede  actual  practice. 

Program  design: 

I)  Ability  to  formulate  own  protocol 
based  on  available  resources,  2) 
Included  technical  documentation  for 
each  recommendation  in  the  guideline, 
and  3)  Structured  as  mandate  to  urge 
compliance. 

Environment:  Program  design: 

1 )  Availability  and  1 )  Administrative  processes  and  forms 

teamwork  of  skilled  and  supported  simple  data  entry,  2)  Staff 

competent  people,  and  2)  developed  their  own  solution  (local 
Caring  ethos  around  being  adaptation  of  the  standard) 
an  employee  translated  not  Engagement: 

only  to  care  of  patients  but  I )  Layering  of  one  information  system 

also  to  care  of  the  over  another  created  dependencies  and 

environment.  forms  of  incentives  for  keeping  the 

quality  of  data  accurate. 
Accountability/leadership: 
Continuous-change  process  where  the 
local  organization  reflected  upon  and 
proactively  reinterpreted  its  own  work 
processes. 
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Chaulagai 
ct  al.  2005 
(18) 


Health  management 
information  system 
(HMIS)  in  Malawi 


Objectives:  5-year  strategy  for 
strengthening  routine  HMIS. 
Revised  procedures  manual 
tested  in  phases  of  18  months 
starting  with  3  health  facilities 
and  progressing  to  district  and 
tertiary  care  facility,  training 
for  members  from  all  districts, 
health  information  policy  and 
indicators  developed,  routine 
monitoring  included. 


Gericke  et 
al.  2005 
(40) 


Objective:  To  illustrate  the  use 
of  a  framework  applied  to 
condom  social  marketing 
programs  for  prevention  of 
HIV/AIDS  and  other  STDs, 
and  DOTS  strategy  for 
tuberculosis  control. 


Study  type:  Descriptive 
study  of  a 

comprehensive  HMIS 
introduced  in  the 
country  in  Jan  2002. 
Analysis:  Not  addressed 


Study  type: 
Development  of  a 
conceptual  framework 
to  categorize 
interventions  according 
to  degree  of  technical 
complexity. 

Analysis:  not  addressed 


Environment:  Accountability/leadership: 

1 )  Desired  teamwork  not  I )  Core  set  of  indicators  purposely 

apparent,  2)  Lack  of  limited  in  number,  2)  Lack  of 

management  knowledge  and  accountability  system  to  address  poor 

skill,  3)  Status  (low)  of  performance. 

personnel  imbalanced  to 

responsibilities,  and  4)  Program  design: 

Operational  pattern  of  I )  Cascade  “training  of  trainers” 

health  sector  not  changed.  approach,  2  half-day  practice-based 

training  sessions  better  than  5  days 
training  in  classroom,  2)  Program 
interpretation  and  dissemination  to 
users  packaged  in  a  variety  of  ready- 
to-use  formats. 

Engagement: 

I )  Focused  on  local  analysis  and  use 
of  information,  and  2)  Lack  of 
financial  incentives  or  reward  system. 

Environment:  Accountability/leadership: 

I )  Human  resources  1 )  Regulation  and  enforcement  need 

insufficient,  2)  Require  for  monitoring,  2)  Need  for  training 

National  and  local  political  and  supervision, 
support,  3)  Pre-existing 

demand  for  program,  4)  Program  design: 

Need  for  regular  supplies,  1 )  Need  for  training  and  supervision, 
equipment  (high  2)  Standardization  and  simplification 

technology,  infrastructure,  of  procedures  allows  services  to  he 
different  types,  provided  by  less  skilled  staff, 

maintenance),  and  5) 

Communication  and  Engagement: 

transport  infrastructure  1 )  Quality  and  quantity  of  non- 

imporlant.  financial  incentives  important,  2) 

Communication  is  important. 
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Kamadjeu 
ct  al.  2005 

(54) 

An  original  cohort 
of  14  providers  (10 
MDs  and  4  nurses) 
from  urban  primary 
health  care  (PHC) 
practice  in 

Cameroon. 

A  locally  designed  EHR 
system  (MEDCAB)  was 
developed  based  on  the 
International  Classification  for 
Primary  Care  (ICPC). 

System  consists  of  several  user 
interfaces  with  multiple 
functionalities.  A  3-day 
training  session  was  organized 
for  prospective  users. 

Williams  et 

al.  2004 
(103) 

Countries  selected 
based  on  availability 
of  descriptive  data; 
included  Tanzania. 
South  Africa, 

Kenya,  Peru  and 
Malawi. 

National  Anlimalarial 
treatment  policy  implemented 
in  country. 

Cutts  et  al. 
1990  (26) 


Urban  and  rural 
areas  of 


Conduction  of  door-to-door 
canvassing  by  volunteers  of 


Study  type:  Cross- 
sectional.  Several  data 
collection  strategies 
identified  using 
subjectivist  approaches, 
but  focused  mostly  on 
direct  observations  and 
interviews. 

Analysis:  Qualitative 
methods 


Study  type:  Case  studies 
summarized  findings 
from  various  malaria 
treatment  policy 
studies.  Included 
review  of  historical 
documents  in  Malawi, 
individual  and  focus 
groups  interviews, 
revie  w  of  historical 
documents,  and 
participatory  methods 
(such  as  time  lines)  in 
other  identified 
countries. 

Analysis:  not  addressed 


Environment: 

I )  Lower  priority  given  to 
project  when  management 
changed,  2)  Limited 
attention  from  funders  and 
policy  makers,  3)  High 
turnover  of  personnel,  4) 
Competition  for  primary 
health  care  data  with  actual 
emergencies,  and  5)  Limited 
computers  and  resources 
available. 


Environment: 

1 )  Broad  lack  of  trust  in 
intervention,  2)  Current 
prescribing  practices 
differed  from  policy  (lack 
of  compliance  with 
regimes),  3)  Political 
changes  affected 
government  stability  and 
support,  and  4)  Felt 
competition  from  other 
national  priorities. 


Program  design: 

1 )  Insufficient  training  for  EHR,  2) 
System  should  be  designed  to  fit  the 
medical  environment,  3)  Decrease  in 
coding  lime  realized  after  4  months. 

Accountability/leadership: 

I)  Experienced  leadership  challenges. 

Engagement: 

I)  Increase  in  users'  self-esteem  and 
positive  image  of  the  facility  in  the 
community  important,  and  2)  Activity 
reports  and  statistics  useful. 
Engagement: 

I)  Implementation  plan  included 
investment  in  health  infrastructure  and 
personnel,  2)  Identification  of 
individuals  to  affect  and  initiate 
change  (gate-keepers),  and  3) 
Presented  arguments  for  change  in 
concert  with  support  from  "credible" 
partners  (international  partners). 

Accountability/leadership: 

1)  Minimal  guidance  on  how  to 
determine  "success"  which  fueled 
insufficient  funding  for 
monitoring/evaluation. 


Study  type:  Coverage  Environment: 

surveys  conducted  in  1)  Extensive  continuing 


Program  design: 

I )  System  designed  to  use  emerging 
lessons. 

Accountability/leadership: 

I )  Expression  of  high  level  interest 
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Mozambique:  Beira,  grassroots  organizations  to  1987  in  provincial 

Tete,  Quelimane,  increase  attendance  to  EPI  capitals  and  in  rural  and 

and  Inhambane.  (Extended  Programme  on  urban  areas  using  WHO 

Immunization).  EPI  cluster  sampling 

method.  Community 
representative  was 
interviewed  in  each 
cluster  before  beginning 
the  household  visits.  A 
detailed  study  of  the 
full  EPI  costs  and  of  the 
mobilizations  cost. 
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donor  support,  2)  Political  and  the  encouragement  to  adopt 

support,  3)  Contribution  of  ambitious  goals  placed  pressure  and 

voluntary  labor,  4)  resulted  in  uncritical  adoption  of 

Resources  were  unavailable,  strategies. 

power  cuts,  5)  Expectation 

of  long  waits,  competing  Engagement: 

demands  on  time  for  users,  I )  Health  staff  established  regular 

6)  Health  services  support,  contacts  with  community,  2) 

improve  patient  flow  in  Communities  shared  responsibility  for 

health  centers.  implementation. 

Program  design: 

1)  Refresher  training  included  in 
design. 


Appendix  G:  List  of  consensus  workshop  participants 


Name 

Position 

Organization 

Robert  Fagan,  BS 

System  Analyst  at  the  Global  Public 
Health  Informatics  Program 

Centers  for  Disease  Control  and 
Prevention 

Henry  Rolka,  RN.  MPS,  MS 

Associate  Director  for  Information 
Exchange 

Centers  for  Disease  Control  and 
Prevention 

David  Blazes,  MD,  MPH 

Chief,  GEIS  Operations  Division 

Armed  Forces  Health  Surveillance 
Center 

Jennifer  Cockrill.  MS,  MPH 

Infectious  Disease  Epidemiologist  in 
the  Division  of  GEIS  Operations 

Armed  Forces  Health  Surveillance 
Center 

Joel  Gaydos,  MD,  MPH 

Science  Advisor 

Armed  Forces  Health  Surveillance 
Center 

Julie  Pavlin,  MD,  PhD, 

MPH 

Deputy  Director 

Armed  Forces  Health  Surveillance 
Center 

Christopher  Perdue.  MD. 
MPH 

Epidemiologist,  Health  Surveillance 
in  the  Division  of  GEIS  Operations 

Armed  Forces  Health  Surveillance 
Center 

Jacqueline  Coberly,  MHS, 
PhD 

Senior  Epidemiologist,  ESSENCE 
Group 

Applied  Physics  Laboratory-Johns 
Hopkins  University 

Sheri  Lewis,  MPH 

Global  Disease  Surveillance  Program 
Manager 

Applied  Physics  Laboratory-Johns 
Hopkins  University 

Richard  Wojcik,  MS 

Lead  Software  Engineer  for  the 
ESSENCE  Project 

Applied  Physics  Laboratory-Johns 
Hopkins  University 

Matthew  Johns,  MPH 

Branch  Chief,  International 
Partnerships 

US  Department  of  Health  and 

Human  Services 

Patrick  Kelley,  MD.  DrPH 

Director,  Board  on  Global  Health 

Institute  of  Medicine 

Ronald  Gimbel,  PhD 

Assistant  Professor,  Biomedical 
Informatics  Department 

Uniformed  Services  University 

Cesar  Munayco,  MD,  MSc, 
MPH 

DrPH  student 

Uniformed  Services  University 

Cecilia  Mundaca,  MD, 

MPH 

DrPH  candidate 

Uniformed  Services  University 
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Appendix  H:  Interview  Introduction  and  Sample  Interview  Guide 

Interview  template  introduction 

Hello.  My  name  is  Cecilia  Mundaca.  Thank  you  for  meeting  with  me  today.  I  am  a  graduate  student  at  the 
Uniformed  Services  University  in  Bethesda,  Maryland.  I  am  here  today  to  learn  about  your  experience 
with  the  implementation  process  of  your  electronic  disease  surveillance  system. 

I  am  doing  this  as  part  of  my  dissertation  work,  which  aims  to  build  a  framework  to  understand  the 
implementation  of  electronic  disease  surveillance  systems  in  Peru.  This  is  not  an  evaluation  of  your 
system  nor  is  it  an  attempt  to  access  your  disease  data.  I  am  only  interested  in  learning  and  understanding 
your  experience  related  to  implementation  of  disease  surveillance  systems  and  your  ideas  about  the 
lessons  you  have  learned  from  such  an  experience. 

1  will  be  asking  you  some  questions  and  taking  notes.  With  your  permission,  I  would  like  to  audio  record 
this  conversation  since  I  don’t  want  to  miss  or  misinterpret  any  information  you  share  with  me.  Our 
conversation  and  this  recording  will  be  confidential  and  will  be  destroyed  after  I  finish  my  work.  Please 
be  sure  you  will  not  be  identified  with  any  information  you  share  nor  will  your  name  be  mentioned  in  any 
publications  or  reports  that  result  from  this  study.  Do  I  have  your  permission  to  record? 

Do  you  have  any  questions  before  we  start? 


215 


Interview  Guide  Template 

Introduction  (See  template) 

Questions: 

Ql .  What  were  some  of  the  reasons  a  decision  was  made  to  implement  this  surveillance  system? 
Q2.  What  factors/stakeholders  helped  in  this  process  to  get  started? 

Probes: 

•  Political  environment 

•  Funding/extemal  support 

•  Existing  infrastructure 

•  Human  resources 

Q3.  What  challenges  did  you  face  in  implementing  the  system? 

Q4.  What  factors  helped  this  system  work? 

Probes: 

•  Political  environment 

•  Funding/extemal  support 

•  Existing  infrastructure 

•  Human  resources 

•  System  design:  training,  tool  design,  program  flexibility 

•  Monitoring  and  evaluation/supervision 

•  Accountability  policies 

•  Stakeholder  involvement 

o  What  stakeholders  were  important  or  critical  in  implementing  the  system? 

•  Communication  strategies 

Q5.  How  do  you  view  this  system  in  terms  of  success? 

•  How  do  you  define  success? 
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Q6.  How  long  did  it  take  for  the  system  to  become  fully  functional? 

•  Can  you  distinguish  any  phases  of  this  process? 

Q7.  Are  there  any  other  successes,  challenges  or  comments  regarding  your  experience  with  the 
implementation  of  an  electronic  surveillance  system  you  would  like  to  share  with  me? 

Q8.  May  I  follow  with  you  if  I  have  further  questions? 
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Interview  notes  template 


Interview  #: 


Date: _ 

Start  time: _ 

End  time: _ 

Audio-recorded?  Y/N  Audio  file  number: _ 

Notes  ,  Observations 


Follow  up  actions/data  sources  identified  Notes  for  review  of  main  points 
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Appendix  I:  Coding  process  example 


Raw  data 

Initial  coding 

Focused  coding 

Theoretical  coding 

Q.  What  are  the  elements  needed  for  a 
successful  implementation? 

I  would  definitely  add  an  adequate  legal 
framework.  When  we  implemented  the 
system  in  Ecuador  in  25  sites,  we  did  our 
part,  train,  supervise,  but  the  Ecuadorian 
military  always  needed  a  legislation  to 
support  the  system.  That  was  never  possible 
to  obtain,  so  the  system  stopped  being 
used... 

Having  a  legal  framework 
is  key  for  these  systems  = 
“Legislation” 

System  with  no  legislation 
was  not  sustainable  = 
“Sustainability” 

“Legislation  as  a 
key  element” 

Having  a  legal 
framework  is  key 
for  sustainability, 
and  requires 
political  will 

“Sustainability 
requires  detailed 
legislation  of  the 
process  of 
implementation  in 
place” 

Q.  Why  was  legislation  not  obtained? 

To  obtain  that  legislation,  they  needed  an 

Political  will  is  needed  to 

“Detailed 

international  agreement  among  organization, 
and  there  was  not  enough  will  from  the  high 
leadership  in  Ecuador  to  have  this  go 
through,  it  is  a  long  and  tedious  process... 

put  in  place  legislation  = 
“Political  will” 

legislation” 

A  description  of 
processes  and 
resources  is  needed 
in  the  legislation 

Q.  What  is  an  adequate  legal  framework? 

Legislation  that  supports  the  implementation 
process  and  a  directive  that  describes  all  the 

Legislation  should  include 
all  the  resources  and 

activities  and  resources  required  to  put  this  in 
place  and  maintain  it... 

This  directive  has  to  be  disseminated  so  all 

processes  required  for  the 
system  to  be  implemented 
and  continue  functioning 

the  surveillance  staff  knows  what  is  the 

=  “Processes  included  in 

purpose  of  this  system 

legislation” 
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Appendix  J:  Results  of  Grey  Literature  review 


Reference 

Country/ 

External 

support 

System/Tool 

Study  design  and 
analysis 

Factors  observed  to 
contribute 

Challenges  and  needs 

Curioso,  W.  H.,  B,  Karras, 
ct  al.  (2005).  "Design  and 
Implementation  of  Cell- 
PREVEN:  A  Real-Time 
Surveillance  System  for 
Adverse  Events  Using 

Cell  Phones  in  Peru." 

AMIA  Svmnosium 
Proceedings:  176-179. 

Country:  Peru 

External  support: 
Imperial  College, 
London 

University  of 

Washington, 

Seattle 

System:  Cell-preven 
(pilot  project,  3  cities) 
Reporting  of  adverse 
events  to  metronidazole 
used  as  treatment  for 
vaginosis 

Tool  developed  on: 
mobile  phones  and 
website 

Report  of 
surveillance 
outputs:  Sep-Dec 
2004 

Use  a  low-tech  system  to  create 
a  sophisticated  solution. 
Partnership  is  key  to  overcome 
technology  barriers. 

Training  is  one  of  the  key 
factors  in  the  success  of  the 
project. 

Domeika,  M.,  G.  Kligys, 
ct  al.  (2009). 
"Implementation  of  a 
national  electronic 
reporting  system  in 
Lithuania " 

Eurosurveillance  14(13): 

1-6. 

Country: 

Lithuania 

External  support: 
Swedish  Health 
Care  Community 

System:  ULISAS 
Reporting  of  76 
communicable 
diseases(All  10  counties) 

Tool  developed  on: 
Off-line  computer 
software 

Report  of  structure 
and 

implementation 
process:  2000- 
2007 

Partnership  with  external  donor 
was  of  vital  importance 
(communication  and  funding). 
Political  engagement  and 
government  financial  support 
facilitated  by  Lithuanian 
legislation. 

Offline  tool  was  selected  to 
meet  resources  capacity  (no 
Internet  access  at  all  sites). 
Bottom  up  policy  with  process 
starling  al  county  level. 

Staff  at  counties  was  motivated 
to  change  to  new  system. 
Resources  already  in  place 

Pilot  project  before  involving 
the  national  level. 

Need  for  integration  to 
other  systems 

Need  for  better  analysis 
tools 

Ear,  S.  (2012).  "Emerging 
Infectious  Disease 
Surveillance  in  Southeast 

Countries: 
Cambodia  and 
Indonesia 

Indonesia: 

System:  EWORS  (Early 
Warning  Outbreak 

Semi-structured 
interviews  of 
stakeholders 

System  is  nationally  owned 

Limited  skilled  human 

resources. 

Poor  staff  and  human 
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Recognition  System,  1 1 
hospital,  8  provinces) 


Asia:  Cambodia, 

Indonesia,  and  the  US 

Naval  Area  Medical 
Research  Unit  2."  Asian 
Security  8(2):  164-187. 

External  donor: 

US  Naval 

Medical 

Research  Unit  2 

Recognition  System,  1 1 
hospital,  8  provinces) 

Tool  developed: 

Offline  software  on 
computer  to  transmit 
over  the  Internet 

Franco,  L.,  J.  Setzer,  et  al. 

Countries: 

System:  IDSR 

(2006).  Improving 

Tanzania  and 

(Integrated  Disease 

Performance  of  IDSR  at 

Ghana 

Surveillance  and 

District  and  Facility 

Response) 

Levels:  Experiences  in 

External  support: 

Tanzania(12  provinces); 

Tanzania  and  Ghana  in 
Making  IDSR 

PHRPIus 

Ghana)  3  regions) 

Operational,  PHRplus 

Tool: 

Computer  software 
Installation  and  use  of 
radio  calls 

Individual  cell  phones  to 
send  text  messages 

Description  of 
efforts  of  PHRplus 
to  improve 
performance  of 


IDSR 


Supervision  anil  follow-up  are 
critical  for  performance 
improvement. 

Addressing  technical 
determinants: 

-  Creating  guidelines  that 
define  norms  and  forms 

-  Clarifying  standards  for  data 


sis 


-  Creating  technical  capacity  to 
collect,  manage  analyze  and 
communicate  data 

Addressing  Organizational 
determinants: 

-  Clarifying  roles  and 
responsibilities 

-  Budgeting  at  local  levels 

-  Outlining  clear  procedures  for 
tasks 

-  Mechanisms  for  involving 
other  actors 

-  Resolving 

communication/reporting 


resources  management. 
Different  priorities 
between  donors  and 
government. 

Poor  host-donor 
relationships. 

Difficulties  incentivizing 
local-level  reporting. 
Limited  government 
resources  and  financial 
commitment. 

Awareness  of  local 
political,  economic  and 

cultural  issues. _ 

Advocacy  based  on  data 
from  district  and  facility 
staff. 

Engagement  of 
stakeholders  at  all  levels. 
Government  commitment 
to  provide  resources, 
accountability  and 
communication 
mechanisms. 

Understand  the  links 
between  the  surveillance 
strategy  and  the  country 
political  context. 
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Huaman,  M.  A.,  R.  V. 
Araujo-Caslillo,  et  al. 
(2009).  "Impact  of  two 
interventions  on 
timeliness  and  data  quality 
of  an  electronic  disease 
surveillance  system  in  a 
resource  limited  setting 
(Peru):  a  prospective 
evaluation."  BMC  Med 

Country:  Peru 

External  support: 
NMRCD  (Naval 
Medical 

Research  Center 
Detachment) 

System:  Alerta 

DISAMAR 

Tool:  Internet-based 
platform  with  phone 
(IVR)  or  computer  based 
reporting 

Inform  Dccis  Mak  9:  16. 

Jefferson,  H.,  B.  Dupuy,  Country:  French  System:  syndromic 

el  al.  (2008).  "Evaluation  Guiana  surveillance  system  2SE 

of  a  syndromic  FAG  (Surveillance 

surveillance  for  the  early  External  support:  Spaliale  des  epidemies 

detection  of  outbreaks  French  Armed  au  Sein  des  Forces 

among  military  personnel  Forces  Armees  cn  Guyane) 

in  a  tropical  country." 

Journal  of  Public  Health 
30(4):  375-383. 


Tool:  a  computer  in  a 
permanent  medical  unit 
or  a  PDA  on  a  mission 


1 2  week 
prospective 
evaluation  (40 
reporting  units) 

3  groups  randomly 
assigned  (phone, 
visit  or  control 
group) 


Evaluation  using 
CDC  framework 
(qualitative  and 
quantitative 
analysis) 


constraints 

-  Strengthening  supervision 
Addressing  workforce 
determinants: 

-  Bottom-up  capacity  building 

-  Creating  training  capacity 

-  Use  of  job  aids  for  standard 
case  definitions,  data 
interpretation,  etc. 


Regular  phone  call  reminders 
enhanced  timeliness  of  report 
Supervision  did  not  improve 
timeliness. 

Supervision  visits  improved 
data  quality  in  the  clinic  sites 
vs.  ships. 


The  timeliness  of  the  system 
was  one  of  its  major  strong 
points. 

The  system  consistently 
delected  febrile  outbreaks. 


Areas  of  acceptability 
needed  to  improve: 

-  Time  taken  to  use  the 
system  (need  to 
simplify  data  input 
forms) 

-  Actual  use  of  the 
system  by  people  who 
enter  the  data 

Introduction  of  a 
standardized  protocol  to 
respond  to  system  alarms. 
Military  specificities  need 
to  be  integrated  in  the 
evaluation  guidelines. 
Minimize  the  occurrence 
of  hindering  technical 
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John,  T.  J..  R.  Samuel,  et 

Country:  India 

System:  NADHI  (North 

al.  (1998).  "Disease 

Acorn  District  Health 

surveillance  al  district 

Information) 

level:  a  model  for 

10  notifiable  diseases 

developing  countries." 

(one  district  in  the 

Lancet  352(9121):  58-61. 

southern  stale  of  Tamil 
Nadu) 

196  government  and  426 
private  reporting  centers 

Tool: 

All  data  were  entered  in 
a  computer 

Kant.  L.  and  S.  K. 

Country:  India 

System:  IDSP 

Krishnan  (2010). 

(Integrated  Disease 

"Information  and 

External  support: 

Surveillance  Project, 

communication 

World  Bank 

85%  of  all  districts) 

technology  in  disease 

(funding) 

surveillance,  India:  a  case 

CDC  and  WHO 

Tool:  Single  stop  web 

studv."  BMC  Public 

(technical 

portal  (Broadband  and 

Health  10  Sunni  1:  SI  1. 

assistance) 

satellite  network) 

24/7  call  center 

Kebede.  S  .  J.  B. 

Country: 

System:  National  Public 

Galabazi,  el  al.  (201 1 ). 

Rwanda 

Health  Laboratory 

"Strengthening  systems 

System  (34  district 

for  communicable  disease 

External  support: 

hospitals,  385  health 

surveillance:  creating  a 

WHO,  CDC, 

centers) 

laboratory  network  in 

Columbia 

Rwanda."  Health  Res 

University  and 

Description  of  the 
implementation  of 
the  project 


Description  of  the 

surveillance 

system 


Evaluation  of  the 
system  (review  of 
documents  and 
interviews  with 
stakeholders) 


Single  disease  surveillance  was 
unlikely  to  motivate  private 
sector  physicians 
Successful  disease  surveillance 
must  combine: 

-  Remove  the  fear  of 
reporting  on  government 
sector  staff 

-  Obtain  private  sector 
participation 

Inexpensive  system. 

Motivation  of  reporting  staff 
was  maintained  by: 

-  Periodic  visits  from  the 
field  staff 

Occasional  CME  meetings 

-  Supply  of  free  vaccines 
Communication  network 
enhanced  and  frequently  used. 
Ongoing  training  for  reporting 
staff  and  data  managers,  use  of 
communication  network  to 
provide  e-learning  experience, 
appropriate  e-modules 
developed. 

Enhanced  networking  of 
infectious  disease  hospitals. 
Quality  and  timelines  of 

reporting  improved. _ 

Structured  governing 
framework  for  public  health 
surveillance. 

Political  commitment  to 
promote  leadership  for  stronger 
laboratory  capacity. 

Defined  roles  and 


problems. _ 

No  reporting  in  rural 
health  centers  due  to  the 
fear  of  being  blamed  lor 
not  reaching  vaccination 
targets. 

Not  able  to  establish 
etiological  diagnosis. 
There  is  no  system  of 
enforcement. 

In  the  absence  of  action  or 
at  least  feedback,  people 
lose  interest. 


Encourage  private  sector 
reporting. 

Strengthening  skills  for 
data  use  and  analysis  at  all 
levels. 

Enhance  integration  with 
other  national  programs. 
Incorporate  mobile,  text, 
voice,  email  and  fax 
technology  to  the  portal. 


Weaknesses  in  the  general 
infrastructure. 

Insufficient  number  of 
skilled  human  resources, 
Budgetary  constraints. 
Shortage  of  vehicles, 
equipment,  supplies  and 
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Policy  Syst  9:  27. 


Tool:  computer  system 


Lukwago.  L.,  M. 
Nanyunja,  cl  at.  (2012). 
"The  implementation  of 
Integrated  Disease 
Surveillance  and 
Response  in  Uganda:  a 
review  of  progress  and 
challenges  between  2001 
and  2007."  Health  Policy 
and  Planning  28(  I ):  30- 
40. 


Country:  Uganda  System:  IDSR 

(Integrated  Disease 


External  support: 
WHO  AFRO, 
CDC,  USAID 


Surveillance  and 
Response) 

Tool: 

Computer  based 


Evaluation  at  the 
national,  district 
and  health  facility 
levels  using  CDC 
and  WHO 
framework  from 
2001-2007  (multi¬ 
stage  stratified 
sampling) 


responsibilities  for  each  level. 
Coordinated  approaches 
between  technical  and  funding 
partners. 

Collaboration  with  external 
laboratories. 

Use  of  performance  results  in 
advocacy  with  national 
stakeholders. 

Advocacy,  provision  of 
equipment,  training  and 
research  (partner  resources). 


IDSR  improved  surveillance 
and  response  efforts 
(improvement  in  completeness 
and  timeliness  of  reporting, 
gradual  increase  in  outbreaks 
recording,  greater  use  of  data 
collected). 

Multiple  communication 
channels:  radio  call,  telephone, 
fax  and  Internet;  between 
district  and  national  level. 
Increased  technical  support  at 
the  startup  of  IDSR. 

Better  availability  of  equipment, 
reagents  and  supplies. 

Weekly  publication  served  as 
advocacy  and  accountability 
tool. 

IDSR  training  support  and  the 
existence  of  surveillance 
management  information 
system  focal  persons  supported 
by  WHO. 


reagents. 

Use  of  multiple  forms 
from  a  variety  of 
implementing  partners 
resulting  in  duplication  of 
efforts  and  resources. 
Geographic  barriers 
Erratic  supply  of 
electricity  affects 
computer  and  Internet 
access. 

Detined  communication 
channels  between  the 
levels  of  the  health  system. 


Decreased  budgetary 
support  from  government. 
Drop  of  timeliness  of 
reporting  probably  due  to 
new  districts  with 
inadequate  capacity. 
Failure  to  recruit  trained 
personnel  due  to  limited 
funding. 

Low  capacity  to  confirm 
priority  diseases. 

Rapid  increase  in  the 
number  of  health  districts 
in  the  strategy. 
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PHRplus  andN.  1.  f.  M. 
Research  (2006).  Follow¬ 
up  Monitoring  and 
Evaluation  of  Integrated 
Disease  Surveillance  and 
Response  in  Tanzania, 
Funding  by  USAID. 


Country: 

Tanzania 

External  support: 
USAID,  Partners 
for  Health 
Reform  plus 
Project  (PHR 
plus) 


System:  ISDR 
(Integrated  Disease 
Surveillance  and 
Response) 

13  notifiable  diseases 
Tool: 

Computerized  system 


PHRplus  (2006).  Georgia 
Immunization  MIS  and 
Disease  Surveillance 
Reforms:  Achievements, 
Lessons  Learned  and 
Future  Directions,  Funded 
by  USAI 


Country: 

Georgia 

External  support: 
PHR  plus 


System: 

Infectious  Disease 
Surveillance 

Tool:  software 
application 


Follow-up 
evaluation  (Jan- 
Mar  2005)  in  12 
districts  from  8 
regions.  Total  of 
109  health 
facilities  visited. 
Sampling 
framework 


Description  of 
efforts  to  enhance 
surveillance 
system  (IDS 
reforms  in  Imereti 
region  2004-2005) 


Link  between  high  political  and 
financial  commitment  and 

ISDR  performance. _ 

Timeliness  and  completeness  of 
reports  increased  substantially. 
Use  of  data  to  plan  and  monitor. 
Monitoring  and  evaluation  and 
support  required. 

Strong  coordination  and 
communication  with  partners 
and  stakeholders. 

Intense  training  and 
development  of  tools  and  job 
aids. 

National  and  regional  level 
commitment. 

Motivation: 

-  Workers  supported  and 
valued  by  colleagues  and 
supervisors 

-  Workers  who  feel  they  are 

making  a  contribution 
Workers  supported  in 
solving  work-related 
problems _ 

Guidelines  for  health  care 
providers. 

Job  aids  for  district  level. 
Laboratory  reference  manual. 
District  level  training. 
Continuous  supervision  and 
support. 

Monitoring  and  evaluation. 
Developing  a  network  of 
laboratories. 

Improving  human  resource 
management. _ 


Need  to  improve  use  of 
case  investigation  forms. 
Not  regular  conduction  of 
analysis  at  the  facility  and 
district  level. 

Follow  up  on  training  on 
outbreak  preparedness. 
Improve  feedback  to  all 
levels  of  the  system. 
Ongoing  support  over  a 
longer  period  of  time  for 
health  workers. 

Lack  of  resources  and 
funding. 

Limited  time  to  do 
surveillance  tasks. 
Competing  priorities  at  the 
health  facility  level. 


Utilization  of  data 
remained  weak. 
Inadequate  public  funding 
of  surveillance  activities. 
Lack  of  legal  or 
administrative  authority 
for  management  and 
governance  issues. 
Inadequate  financial  and 
administrative  motivators 
for  reporting  staff. 

Donors  should  demand 
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Rajatonirina,  S..  J.  M.  Country: 


Heraud,  et  a).  (2012). 
"Short  message  service 
sentinel  surveillance  of 
influenza-like  illness  in 
Madagascar.  2008-2012." 
Bull  World  Health  Organ 


90:  385-389. 


Madagascar 

External  support: 
Inslitut  Pasteur 
of  Madagascar 


System:  Sentinel 
Surveillance  system  for 
Influenza  like  illness  (34 
sites) 

Tool:  use  of  cell  phones 


Robertson.  C.,  K. 

Sawford,  el  al.  (2010). 
"Mobile  phone-based 
infectious  disease 
surveillance  system,  Sri 
Lanka."  Emerging 
infectious  diseases  16(10): 


1524-1531. 


Country:  Sri 
I^anka 

External  support: 
Teasdale-Corti 
Global  Health 
Partnership  and 
the  National 


System:  Infectious 
Disease  Surveillance  and 
Analysis  System 
(IDSAS).  Pilot  project  in 
4  districts 

Tool:  Mobile  phone 
based 


Improvements  need  to  reflect  governments  share  a 
national  high  priorities.  portion  of  the  costs  which 

Appropriate  organizational  improves  the  chances  for 

structure.  sustainability. 

Sufficient  infrastructure.  Overall  strategic  vision 

Financial  structures.  prior  to  reforming  system 

Routine  program  of  continuous  components, 

education  covering  surveillance  Adequate  staffing  with 
topics.  clearly  defined  roles. 

In  a  small  country,  disease  Finalize  development  of 

surveillance  funding  should  public  health  law. 
come  from  the  central  budget.  Advocacy  measures  help 
Disease  surveillance  projects  identify  and  clarify 
should  take  two  years  for  priorities, 

development  and  testing  and  Integrate  individual 

three  years  for  adequate  program  specific  software 

institutionalization. _  into  a  unified  platform. 

Evaluation  of  the  Advantage  of  suing  syndromic  Need  to  rely  on  strong 

implementation  data  for  timeliness  in  outbreak  communication  systems, 
process  (2008-  detection.  cost  of  maintaining  this 

101 1 )  Mobile  phone  technology  network  needs  to 

proved  to  be  a  useful  tool.  estimated. 

Motivation  of  reporting  staff:  High  staff  turnover. 

Provision  of  medical  equipment 
Training  opportunities  (annual 
epidemiology  workshop  and 
annual  meeting) 

Health  district  officers  trained 

_ to  train  and  supervise  new  staff. _ 

Description  of  the  Technical  considerations:  Distinction  between  the 

design  and  Use  of  familiar  technologies  IDSAS  as  a  research 

implementation  minimize  training  time  project  and  the  national 

Developing  local  expertise  to  reporting  system, 

ensure  sustained  technical  and 
logistical  support 
Financial  considerations: 

Hardware  available  should  be 
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Sciences  and 
Engineering 
Research 
Council  of 
Canada 


Singh,  V.,  R. 
Madhusudan,  et  al, 

(201 1 ).  "An  evaluation  of 
mobile  phone  technology 
use  for  Integrated  Disease 
Surveillance  Project 
(IDSP)  in  Andhra 
Pradesh.  India."  ISPS 
Conference  Abstracts. 


Siswoyo,  H.,  M.  Permana, 
et  al.  (2008).  "EWORS: 
using  a  syndromic-based 
surveillance  tool  for 
disease  outbreak  detection 
in  Indonesia."  BMC  Proc 
2(S3). 


Country:  India 
(6  districts  in 
Andhra  Pradesh) 


Country: 

Indonesia 

External  support: 
Naval  Medical 
Research  Unit  2 
(NAMRU  2) 


System:  IDSP 
(Integrated  Disease 
Surveillance  Project) 

Tool:  use  of  mobile 
phones 


System:  EWORS  (Early 
Warning  Outbreak 
Response  System) 

29  signs  and  symptoms 
compatible  with 
infectious  diseases 
(9  hospitals) 


Evaluation  of  the 
system 
(probability 
proportion  to  size 
sampling  strategy  ) 


Description  of  the 
functioning  of  the 
system 


used 

Open  source  software  should  be 
selected 

Political  considerations  (most 
important  factor): 

External  financial  support  to 
cover  initial  investment 
required 

Support  at  all  levels  of  the 
government  al  the  early 
implementation  phase 
In  the  design  process:  select 
outputs  of  the  system  and  their 
value 

Ethical,  societal,  cultural: 

Build  appropriate  data  security 
in  the  system 
Adoption  requires  user 
acceptance  and  new  technical 
skills  (time  and  experience) 


Use  of  mobile  phone 
technology  has  the  potential  to 
enhance  efficiency  and 
effectiveness  of  the  system. 


Human  resources:  hospital 
program  manager,  data  entry 
clerk  exclusively  dedicated. 
Regular  basis  training  of  health 
care  workers. 

Hospital  directors’ 
commitment. 

EWORS  package. 


Lack  of  clear  directives  for 
implementation. 

Lack  of  guidelines  for 
usage. 

Lack  of  systematic 
training  of  workforce  for 
using  a  system. 

Adequate  resources. 


High  cost  to  continue  the 
program. 

Difficulties  in  creating 
team  work. 

Struggle  to  maintain 
commitment  by 
participating  hospitals. 
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Solo,  G„  R.  V.  Araujo- 
Caslillo,  et  al.  (2008). 
"Challenges  in  the 
implementation  of  an 
electronic  surveillance 
system  in  a  resource- 
limited  setting:  Alerta,  in 
Peru."  BMC  Proc  2  Suppl 
3:  S4. 


Country:  Peru 

External  support: 
Naval  Medical 
Research  Center 
Detachment 
(NMRCD) 


Tool:  hospital-based 
network  of  computerized 
linkages 


System:  Alerta  (45 
clinical  diagnosis  or 
syndromes),  88  navy 
sites  and  120  army  sites 

Tool:  software  access 
through  any  public 
phone  or  computer  with 
Internet  connectivity 


Retrospective 
description  of  the 
challenges  in  the 
implementation 


Finances. 

Computer  networking. 
Technical  aspects:  special 
phone  lines,  dedicated 
computer,  data  collection  forms. 
Data  analysis  results. 


Use  of  resources  already  in 
place. 

Training  before  deployments 
and  local  replication  of  courses. 
VHF  radio  relay  process  that 
reports  from  remote  areas. 
Paired  surveillance  sites  with 
regional  diagnostic  laboratories 
and  streamline  process  of 
sending  samples  to  lab  in  the 
capital. 

Use  of  standardized  templates 
and  reporting  via  electronics 
means. 

Short  average  time  to  send  a 
report. 

Standard  analysis  is  automated 
to  facilitate  use  of  data. 
Continuous  monitoring  of  the 
sites  for  timeliness  and  error 
rates  allowed  targeted  training. 
Convincing  stakeholders  that 
their  efforts  are  useful  and 
valuable. 

Creating  a  surveillance  culture 
by  training. 

Fully  dedicated  technician  to 
monitor  the  sites. 

Persistent  and  focused 
evaluation. 


Issues  with  stakeholders: 
Limited  epidemiological 
training 

Continuous  movement  of 
trained  personnel 
Resources: 

Remote  areas  without 

access  to  phones  and 

computers 

Lack  of  laboratory 

confirmation 

Processes: 

Reporting  a  large  number 
of  cases 

Low  report  on  lime  rate, 
low  data  quality  and  low 
outbreak  detection  during 
expansion  process 
Central  hub: 

Data  analysis  takes  long 
time  to  perform 
Expansion  of  new  sites 
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Sow.  I.,  W.  Alemii.  el  al. 
(2010).  "Trained  District 
Health  Personnel  and  The 
Performance  of  Integrated 
Disease  Surveillance  in 
the  WHO  African 
Region."  East  African 
Journal  of  Public  Health 


7(1):  16-19. 


Country:  8 


System:  IDSR 


African  countries  (Integrated  Disease 
Surveillance  and 


External  support: 
WHO.  CDC 


Response) 

19  priority  diseases 


Wamala,  J.  F.,  C.  Okot,  el  Country:  Uganda  System:  IDSR 


al.  (2010).  "Assessment  of 
core  capacities  for  the 
International  Health 
Regulations  (IHR[2005))- 
-Uganda,  2009."  BMC 
Public  Health  10  Suppl  I : 
S9. 


External  support: 
WHO 


(Integrated  Disease 
Surveillance  and 
Response) 

Tool:  SMS,  telephone, 
email  and  radio  call 


Weber,  I.  B.  (2007). 
Evaluation  of  the 
Notifiable  Disease 
Surveillance  System  in 
Gauteng  Province,  South 
Africa,  University  of 
Pretoria.  Master  of 


Country: 
South  Africa 


System:  Notifiable 
disease  surveillance 
system  (Gauteng 
Province) 

Tool:  electronic  data 
submission 


Secondary 
analysis  of  the 
evaluation  reports 
to  asses  training 
approaches 


Descriptive  cross- 
sectional 

assessment  of  IHR 
within  IDSR 
(convenience 
sample) 


Training  of  district  health 
personnel  contributed  to  the 
improvement  in  timeliness  and 
completeness  of  reporting. 
Sustained  commitment  of  health 
facility  personnel. 

Sustainable  district  supportive 
supervision  and  feedback  may 
sustain  the  performance. 
Reliable  communication 
Availability  of  simplified 
reporting  tools. 

Availability  of  sustained  and 
integrated  funding  of  training 
activities. 


Evaluation  study 
(qualitative  and 
quantitative 
analysis)  Jan-Jun 
2006 


Maintain  communication  and 
feedback  on  a  regular  basis. 
Evaluation  of  the  system  at 
regular  intervals. 

Prioritize  list  of  diseases  under 
surveillance. 

Planning  for  human  resources. 


Reaching  the  health 
facility  level  in  3-4  years 
after  implementation. 
Single  day-long  training 
sessions  are  inadequate. 
Lack  of  simplified 
surveillance  forms  impact 
the  performance. 


Revision  of  national 
legislation  facilitated  full 
implementation. 
Partnerships  and  inter¬ 
sectoral  collaboration. 
Advocacy  for  IHR  needs 
to  be  prioritized  to  ensure 
commitment  of  resources. 
Revised  tools  and 
guidelines  should  be 
disseminated  to  all 
stakeholders. 

Build  human  capacity. 
Development  of  a  national 

preparedness  plan. _ 

Low  level  of  compliance 
due  to  time  constraints  and 
lack  of  motivation. 

Limited  knowledge  of  the 
notification  status  of 
selected  medical 
conditions. 
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Health:  106. 


WHO  (2003) 

Implementing  integrated 
disease  surveillance  and 
response  -  Successes, 
challenges  and  lessons 
learned:  the  Uganda 
experience  (ISDR). 

Wceklv  enidemioloeical 
record  27. 229-240 

Country:  Uganda 
External  support: 
WHO,  CDC, 
USAID,  UN 

System:  IDSR 
(Integrated  Disease 
Surveillance  and 
Response) 

Tool:  electronic 
communication  between 
central  level  and 
districts,  and  radio 
contact. 

Documentation  of 
the 

implementation 


Training  and  support  of  reports 
in  both  public  and  private 
sector. 

Incorporation  of  notifiable 
disease  reporting  in  CME 
programs. 

Regular  bidirectional 
communication  should  be 
maintained  between  public  and 
private  sectors. 

Dissemination  of  results  and 
interpretation  of  notification 

data. _ 

Strong  support  from  the 
political  leadership. 

The  MOH  provides  continuous 
feedback  to  all  levels  on  ISDR. 
Development  of  joint 
surveillance  work  plans  that 
articulate  roles  and 
responsibilities  of  each 
stakeholder. 

Creation  of  a  budget-line  for 
surveillance. 

The  effectiveness  of  outbreak 
control  through  the  system 
increased  political  commitment. 
National  IDSR  committee 
created  to  coordinate 
surveillance  activities.  Team 
spirit  and  trust  grew  among 
stakeholders. 

Standard  case  definitions  and 
actions  thresholds  for  priority 
diseases  were  distributed  to  all 
facilities. 

Award  prizes  to  districts  with 


Lack  of  feedback  between 
the  public  and  private 
sectors. 

Lack  of  standardized 
procedures  of  reporting. 
The  quality  of  notification. 
No  dedicated  surveillance 
officer  at  the  provincial 
unit. 

High  workload  on  a 
limited  number  of  staff. 


Need  to  improve  feedback 
from  district  to  lower  elves 
and  motivate  health 
workers. 

The  private  sector  has  not 
been  fully  involved 
(framework  needs  to  be 
developed). 

Data  management  and 
analysis  at  lower  levels. 
Critical  need  for  training 
in  statistical  and  mapping 
software  programs. 

Build  district  capacity  in 
computerized  data 
management  systems. 
Standardization  of 
surveillance  tools  and 
software. 

Networking  between 
laboratory  and 
epidemiological 
surveillance. 

Need  to  strengthen 
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WHO  (2005).  Global 

Countries: 

National  disease 

consultation  on 

Uganda  1, 

surveillance  systems 

strengthening  national 

India2, 

capacities  for  surveillance 

Romania3, 

Tool:  computerized  data 

and  control  of 

I-ebanon4, 

reporting  system7 

communicable  diseases. 

Cambodia5, 

Geneva,  World  Health 

Ghana 

Organization. 

andTanzania6, 

Brazil7 

3-day  global 
workshop  on 
strengthening 
national  capacities 
for  surveillance 
and  control  of 
communicable 
diseases 


good  performance. 

Use  of  surveillance  information 
to  formulate  policy. 

Weekly  newsletter  is  a  tool  for 
monitoring  performance  by 
politicians,  resulting  in 
increased  commitment. 
Simplified  forms. 

Supervision  relevant  to  the 
work  of  reporting  staff. 

Creation  of  laboratory 
networks,  role  of  lab  personnel 
well  defined. 

Use  of  outbreak  awareness  to 
increase  commitment. 

WHO’s  role  was  critical  in 
technically  supporting  the 
process. 

Advocacy  and  networking  are 
crucial. 

Strong  donor  support. _ 

High  level  commitment  1,3,4 
(Development  of  National 
Health  Sector  Strategic  Plan6, 
focused  district  plans  and 
resources2;  national 
eommiitee2)i 
Regular  feedback  1,4 
Local  capacity  buildingl.4 
Networking4.7  and  partnership3 
(laboratory)  1,7 
Training  1,6, 7.  training 
manuals6 

Behavioral  change  was  critical 
for  application  of  skills  learned6 
Clearly  communicate 
expectations  of  performance6 


laboratory  capacity. 
Create  a  mass  of  skilled 
human  resources. 


Analysis  at  lower  levels 
Support  from  all 
stakeholders! 

Improve  the  quality  of  data 
and  use  for  public  action  I 
Filling  gaps  in 
coordination3 
Eslablishing/updaling  a 
legal  framcwork3,4,5 
Lack  of  accreditation  and 
laboratory  quality 
assurance3 

Raise  awareness  of  health 
providers4 
Maximize  use  of 
information  teehnology4 
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WHO,  CDC,  ct  al.  (2003).  Countries:  System:  IDSR 

The  Implementation  of  Burkina  Faso,  (Integrated  Disease 

Integrated  Disease  Ethiopia,  Ghana.  Surveillance  and 

Surveillance  and  Mali,  Uganda,  Responses)  and 

Response  in  the  African  Southern  Soudan  EWARN  (Early 
and  Eastern  Warning  and  Response 

Mediterranean  Regions.  Nov-Dec  2002  Network) 

External  donor: 

WHO 


Wuhib,  T„  T.  L.  Chorba,  Country:  System:  Armenian 

et  al.  (2002).  "Assessment  Republic  of  Infectious  Disease 

of  the  infectious  diseases  Armenia  Surveillance  .System 

surveillance  system  of  the  (AIDSS) 

Republic  of  Armenia:  an  External  support  64  infectious  diseases 
example  of  surveillance  in  USAID,  CDC 
the  Republics  of  the 
former  Soviet  Union." 


Appropriate  job  aids,  tools, 
guidelines6 

Social  mobilization  1 

Monitoring  and  evaluation! 

Use  of  thresholds  1 ,4 

Legal  frame  work 4 

Producing  regular 
updates5 

Improving  data 
management  at  central 
Icvel5 

Update  of  list  of  diseases5 

Documentation  of 

Sensitization  efforts  on  all 

Bring  guidelines  to  the 

the 

stakeholders  (direct 

peripheral  levels. 

implementation  of 

communication  with  health 

Continuous  sensitization 

these  systems 

workers  with  relevant 

on  basic  principles  of 

(mainly  qualitative 

information). 

surveillance  and  the  use  of 

data) 

Adaptation  of  guidelines  and 

adapted  tools  at  all  levels. 

tools. 

Limited  access  to  remote 

Training  sessions. 

communities  due  to  poor 

Regularly  supervise  the  trained 

transportation  network  and 

personnel. 

absence  of  phone 

Informal  opportunities  for  on- 

communication. 

site  training. 

Integration  should  be  at 

Evaluate  the  performance  of  the 

the  donor  level,  national 

system. 

programs  and  regional 

Meetings  of  surveillance  and 

offices. 

lab  personnel. 

Improve  technical  level 

Use  of  radio  network  is 

coordination  and 

effective  to  bridge  the 

integration. 

communication  gap. 

Strengthen  links  between 

Coordination  with  other 

central,  regional  and 

programs. 

Directive  at  the  national  level. 

district  labs. 

Evaluation  of  the 

Eliminate  punitive 

List  of  diseases  under 

system  using  CDC 

consequences  for  reporting. 

surveillance  had  not  been 

framework 

Prioritize  diseases  under 

prioritized. 

(qualitative 

surveillance  based  on  public 

No  tiered  case  definitions 

methods) 

health  importance. 

(confirmed,  probable. 

Categorized  diseases  into  a 

suspected). 

tiered  case  definition. 

Epidemiologist  did  not  use 

Simple  reporting  procedures 

analytic  methods  to  assess 
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and  forms. 

Provided  ongoing  training  in 
analytic  epidemiology. 

Base  interventions  on 
epidemiologic  evidence. 

Large  numbers  of  health 
facilities  and  trained  personnel. 
Separation  of  preventive  from 
curative  medicine. 

Provide  feedback  to  all 
reporting  sources  and  share 
information  across  programs. 


risk  factors  for  diseases. 
Little  or  no  feedback 
provided  to  original 
reporting  sources. 

Data  collection  process 
was  complex  (labor 
intensive). 

Constrained  resources. 
Limited  lab  confirmation 
No  outbreak  reporting  due 
to  fear  to  punishment. 
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Appendix  K:  Workshop  agenda 


Expert  Consensus  Workshop  on  Electronic  Disease  Surveillance  Systems  Implementation  in 

Developing  Countries 
AGENDA 

_ _ Wednesday  February  22nd,  2012  _ 


08:00-08:15 

Welcome  and  Opening  Remarks 

Dr.  Julie  Pavlin 

08:15-08:30 

Introduction  of  participants 

08:30-09:15 

Presentation  of  literature  results 

Objectives  of  the  Workshop 

Dr.  Cecilia  Mundaca 

09:15-10:00 

Overview  of  common  vision 

General  discussion:  Electronic  disease  surveillance 
definition 

Group  assignments 

Dr.  Patrick  Kelley 

10:00-10:15 

Break 

10:15  -12:00 

Session  1 :  Group  discussion 

Group  1 :  Phases  or  stages  of  implementation  of  an 
electronic  disease  surveillance  system  in  developing 
countries  (Metrics) 

Group  2:  Factors  related  to  successful  implementation 
of  electronic  disease  surveillance  systems  in 
developing  countries 

Group  1: 

Dr.  Jacqueline  Coberly 

Dr.  Ron  Gimbel 

Group  2: 

Sheri  Lewis 

Dr.  Julie  Pavlin 

12:00-  13:30 

Lunch 

Optional  tour  to  AFHSC  facilities 

Dr.  Joel  Gaydos 

13:30-  14:00 

Inter-group  sharing 

Dr.  Cecilia  Mundaca 

Sheri  Lewis 

Dr.  Jacqueline  Coberly 

14:00-  15:30 

Session  2:  Group  discussion  (continuation) 

Group  1  &  Group  2 

Group  I: 

Dr.  Jacqueline  Coberly 

Dr.  Ron  Gimbel 

Group  2: 

Sheri  Lewis 

Dr.  Julie  Pavlin 

15:30-  16:30 

Inter-group  sharing 

End  of  workshop  debriefing 
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Appendix  L:  List  of  interviewees 


Interview  # 

Organization 

Stakeholder 

#  Interviewees 
Military(M)/  CivilianfC) 

1 

NAMRU-6 

Technical  assistance 

C=2 

2 

Peruvian  Navy 

National  surveillance  team 

M=1 

3 

Peruvian  Army 

National  surveillance  team 

C=1 

4 

Peruvian  Army 

National  surveillance  team 

C=1 

5 

MOH-  NIH 

National  surveillance  team 

C=1 

6 

Peruvian  Navy 

National  surveillance  team 

M=1 

7 

Voxiva 

Technical  assistance 

C=1 

8 

MOH-DGE 

National  surveillance  team 

C=1 

9 

MOH-DGE 

National  surveillance  team 

C=1 

10 

MOH-NIH 

National  surveillance  team 

C=I 

11 

Peruvian  Army 

National  surveillance  team 

C=1 

12 

NAMRU-6 

Technical  assistance 

C=3 

13 

Peruvian  Navy 

National  surveillance  team 

C=1  M=1 

14 

Peruvian  Navy 

Reporting  staff 

C=l  M=3 

15 

Peruvian  Army 

National  surveillance  team 

C=l 

16 

Peruvian  Navy 

Reporting  staff 

M=5 

17 

Peruvian  Army 

Reporting  staff 

C=1 

18 

Peruvian  Army 

Reporting  staff 

C=l 

19 

Peruvian  Navy 

Reporting  staff 

C=1 

20 

MOH 

Administrative  level 

Ol 

21 

Peruvian  Army 

National  surveillance  team 

C=1 

22 

MOH 

Reporting  staff 

C=1 

23 

MOH 

Regional  surveillance  team 

C=1 

235 


Appendix  M:  Glossary 


Purposeful  sampling:  Purposeful  sampling  focuses  on  selecting  information-rich  cases  whose 
study  will  illuminate  the  questions  under  study.  Purposeful  sampling  is  sometimes  called 
purposive  or  judgment  sampling:  “In  judgment  sampling,  you  decide  the  purpose  you  want 
informants  (or  communities)  to  serve,  and  you  go  out  to  find  some  (Bertrand  2000:  176).  There 
are  several  different  strategics  for  purposefully  selecting  information-rich  cases.  The  logic  of  each 
strategy  serves  a  particular  purpose.  Source:  Patton  M.  Qualitative  Research  &  Evaluation 
Methods  3  edition  2002. 

Note:  Purposive  sampling  is  not  haphazard.  Purposive  sampling  has  a  purpose:  a  subgroup  has 
been  identified  and  a  rationale  has  been  developed  to  study  them.  Convenience  sampling  is 
haphazard  sampling.  Data  are  collected  from  those  who  happen  to  be  around.  If  we  were  to  select 
the  first  four  people  coming  into  our  agency  for  our  sample,  regardless  of  who  they  were,  we 
would  be  drawing  a  convenience  sample.  There  are  reasons  why  these  people  come  at  this  time 
and  we  do  not  know  those  reasons.  Therefore,  the  sample  is  neither  representative  nor  purposive. 
Source:  Fitzpatrick  J.  Program  Evaluation  -  Alternatives  Approaches  and  Practical  Guidelines.  4th 
edition  201 1. 

Criterion  sampling:  The  logic  of  criterion  sampling  is  to  review  and  study  all  cases  that  meet  some 
predetermined  criterion  of  importance,  a  strategy  common  in  quality  assurance  efforts.  Source: 
Patton  M.  Qualitative  Research  &  Evaluation  Methods.  3rd  edition  2002. 

Snowball  sampling:  This  is  an  approach  for  locating  information-rich  key  informants  or  critical 

cases.  The  process  begins  by  asking  well-situated  people:  “Who  knows  a  lot  about _ ?  Whom 

should  I  talk  to?”  By  asking  a  number  of  people  who  else  to  talk  with,  the  snowball  gets  bigger 
and  bigger  as  you  accumulate  new  information-rich  cases.  Source:  Patton  M.  Qualitative  Research 
&  Evaluation  Methods  3rd  edition  2002. 

Theoretical  sampling:  A  method  of  data  collection  based  on  concepts/themes  derived  from  data. 
The  purpose  of  theoretical  sampling  is  to  collect  data  from  places,  people,  and  events  that  will 
maximize  opportunities  to  develop  concepts  in  terms  of  their  properties  and  dimensions,  uncover 
variations,  and  identify  relationships  between  concepts.  Source:  Corbin  J,  Strauss  A.  Basics  of 
Qualitative  Research  3rd  edition  2008. 

Intensity  sampling:  An  intensity  sample  consists  on  information-rich  cases  that  manifest  the 
phenomenon  of  interested  intensely  (but  not  extremely).  Extreme  or  deviant  cases  may  be  so 
unusual  as  to  distort  the  manifestation  of  the  phenomenon  of  interest.  Using  the  logic  of  intensity 
sampling,  one  seeks  excellent  or  rich  examples  of  the  phenomenon  of  interest,  but  not  highly 
unusual  cases.  Source:  Patton  M.  Qualitative  Research  &  Evaluation  Methods  3  edition  2002. 
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Appendix  N:  Informed  Consent 


WHY  ARE  YOU  BEING  INVITED  TO  TAKE  PART  IN  THIS  RESEARCH? 

You  are  being  invited  to  take  part  in  a  research  study  about  implementation  of  electronic 
disease  surveillance  systems  in  Peru  and  the  elements  related  to  the  success  of  this 
process.  You  are  being  asked  to  take  part  in  this  research  due  to  your  experience  in  such 
process.  If  you  volunteer  to  take  part  in  this  study,  you  will  be  one  of  about  50  people  to 
do  so. 

WHO  IS  DOING  THE  STUDY? 

The  person  in  charge  of  this  study  is  Dr.  Carmen  Mundaca  who  is  a  doctoral  student  at 
the  Uniformed  Services  University.  This  study  is  part  of  her  dissertation  work.  This  work 
will  be  supervised  by  her  thesis  advisory  committee  composed  by  six  professors. 

WHAT  IS  THE  PURPOSE  OF  THIS  STUDY? 

The  study  seeks  to  understand  how  electronic  diseases  surveillance  systems  can  be 
successfully  implemented  in  Peru.  Participants  will  be  asked  to  share  their  experiences 
with  successes  and/or  challenges  when  working  on  implementing  such  systems.  We 
expect  that  the  results  of  this  study  will  potentially  guide  development  of  a  framework  for 
successful  implementation  of  such  systems  in  similar  settings. 

WHERE  IS  THE  STUDY  GOING  TO  TAKE  PLACE  AND  HOW  LONG  WILL  IT 
LAST? 

The  research  will  be  conducted  in  Peru;  potentially  we  will  be  interviewing  participants 
from  the  Ministry  of  Health  and  Armed  Forces.  The  total  amount  of  time  you  will  be 
asked  to  volunteer  for  this  study  is  approximately  60  minutes.  There  might  be  follow-on 
activity  for  some  participants  if  during  the  analysis  the  researcher  needs  to  clarify 
concepts  or  obtain  any  additional  information  on  what  was  shared  during  the  interview. 
Participants  will  be  asked  during  the  interview  if  they  agree  to  be  contacted  if  needed.  If 
the  participant  agrees,  they  will  be  contacted  by  email  and  a  conversation  will  be 
arranged  either  by  phone  or  Skype. 

WHAT  WILL  YOU  BE  ASKED  TO  DO? 

You  will  be  asked  six  to  seven  open-ended  questions.  The  interview  is  expected  to  last 
approximately  60  minutes.  You  will  be  provided  with  an  introduction  with  the  purpose  of 
the  study  and  if  you  agree,  the  interview  will  be  audio-recorded  and  transcribed  verbatim. 
Nothing  you  share  will  be  linked  directly  to  you.  At  the  end  of  the  interview,  the 
researcher  will  provide  you  with  a  summary  of  the  key  points  you  shared  during  the 
interview  to  allow  you  the  opportunity  for  any  clarification  or  additional  comments  as 
necessary. 
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ARE  THERE  REASONS  WHY  YOU  SHOULD  NOT  TAKE  PART  IN  THIS 
STUDY? 


You  do  not  have  to  participate  in  this  study  if  you  choose  not  to.  Your  decision  to 
participate  or  not  to  do  it  will  not  be  shared  with  your  superior. 

WHAT  ARE  THE  POSSIBLE  RISKS  AND  DISCOMFORTS? 

There  are  no  anticipated  risks  associated  with  this  study 

WILL  YOU  BENEFIT  FROM  TAKING  PART  IN  THIS  STUDY? 

There  is  no  guarantee  that  you  will  get  any  benefit  from  taking  part  in  this  study.  Your 
participation  might  provide  useful  information  to  shape  the  design  of  a  framework  to 
guide  the  implementation  of  electronic  disease  surveillance  systems  in  similar  settings. 

DO  YOU  HAVE  TO  TAKE  PART  IN  THE  STUDY? 

If  you  decide  to  take  part  in  the  study,  it  should  be  because  you  want  to  volunteer.  You 
will  not  lose  benefits  or  rights  you  would  normally  have  if  you  choose  not  to  participate. 
You  can  stop  at  any  time  during  the  study  and  still  keep  the  benefits  and  rights  you  had 
before  volunteering.  If  you  decide  not  to  take  part  in  this  study,  your  decision  will  not  be 
shared  with  anyone. 

IF  YOU  DON’T  WANT  TO  TAKE  PART  IN  THE  STUDY,  ARE  THERE  OTHER 
CHOICES? 

If  you  do  not  want  to  be  in  the  study,  there  are  no  other  choices  except  not  to  take  part  in 
the  study. 

WHO  WILL  SEE  THE  INFORMATION  THAT  YOU  GIVE? 

We  will  keep  private  all  research  records  that  identify  you  to  the  extent  possible.  Your 
name  will  not  be  associated  with  the  interview  notes  or  the  interview  transcript.  Your 
information  will  be  combined  with  information  from  other  participants  taking  part  in  the 
study.  We  are  likely  to  publish  the  results  of  this  study;  however,  you  will  not  be 
identified  in  these  written  materials. 

WILL  YOU  RECEIVE  ANY  REWARDS  FOR  TAKING  PART  IN  THIS  STUDY? 

You  will  not  receive  any  rewards  or  payment  for  taking  part  in  the  study. 

WHAT  IF  YOU  HAVE  QUESTIONS,  SUGGESTIONS,  CONCERNS,  OR 
COMPLAINTS? 
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Before  you  decide  whether  to  accept  this  invitation  to  take  part  in  the  study,  please  ask 
any  questions  that  might  come  to  mind  now.  Later,  if  you  have  questions,  suggestions, 
concerns,  or  complaints  about  the  study,  you  can  contact  Dr.  Carmen  Mundaca  at 
cmundaca@usuhs.mil  (011)  511-  571-268-8366.  If  you  have  any  questions  about  your 
rights  as  a  volunteer  in  this  research,  contact  the  staff  in  the  Office  of  Research  at  the 

usu. 

WHAT  ELSE  DO  YOU  NEED  TO  KNOW? 

USU  is  providing  financial  support  and/or  material  for  this  study. 


Signature  of  person  agreeing  to  take  part  in  the  study  Date 

Printed  name  of  person  agreeing  to  take  part  in  the  study 


Name  of  [authorized]  person  obtaining  informed  consent  Date 

Signature  of  Investigator 


Name  of  witness  to  informed  consent  Date 


Signature  of  Witness 
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